Virtual reality laparoscopic tools

ABSTRACT

A virtual reality system providing a virtual robotic surgical environment, and methods for using the virtual reality system, are described herein. The virtual reality system may simulate a robotic surgical environment in which a user may operate both a robotically-controlled surgical instrument using a handheld controller and a manual laparoscopic surgical instrument while adjacent a patient table. For example, the virtual reality system f may include one or more processors configured to generate a virtual robotic surgical environment comprising at least one virtual robotic arm and at least one virtual manual laparoscopic tool, a first handheld device communicatively coupled to the virtual reality controller for manipulating the at least one virtual robotic arm in the virtual robotic surgical environment, and a second handheld device comprising a handheld portion and a tool feature representative of at least a portion of a manual laparoscopic tool, wherein the second handheld device is communicatively coupled to the virtual reality controller for manipulating the at least one virtual manual laparoscopic tool in the virtual robotic surgical environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/018,012, filed on Jun. 25, 2018, which claims priority to U.S. PatentApplication Ser. No. 62/526,896, filed Jun. 29, 2017, the contents ofwhich are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

This invention relates generally to the field of robotic surgery, andmore specifically to new and useful systems and methods for providingvirtual robotic surgical environments.

BACKGROUND

Minimally-invasive surgery (MIS), such as laparoscopic surgery, involvestechniques intended to reduce tissue damage during a surgical procedure.For example, laparoscopic procedures typically involve creating a numberof small incisions in the patient (e.g., in the abdomen), andintroducing one or more surgical instruments (e.g., an end effector, atleast one camera, etc.) through the incisions into the patient. Thesurgical procedures may then be performed using the introduced surgicalinstruments, with the visualization aid provided by the camera.

Generally, MIS provides multiple benefits, such as reduced patientscarring, less patient pain, shorter patient recovery periods, and lowermedical treatment costs associated with patient recovery. In someembodiments, MIS may be performed with robotic systems that include oneor more robotic arms for manipulating surgical instruments based oncommands from an operator. A robotic arm may, for example, support atits distal end various devices such as surgical end effectors, imagingdevices, cannulae for providing access to the patient's body cavity andorgans, etc.

Robotic surgical systems are generally complex systems performingcomplex procedures. Accordingly, a user (e.g., surgeons) generally mayrequire significant training and experience to successfully operate arobotic surgical system. Such training and experience is advantageous toeffectively plan the specifics of MIS procedures (e.g., determineoptimal number, location, and orientation of robotic arms, determineoptical number and location of incisions, determine optimal types andsizes of surgical instruments, determine order of actions in aprocedure, etc.).

Additionally, the design process of robotic surgical systems may also becomplicated. For example, improvements in hardware (e.g., robotic arms)are prototyped as physical embodiments and physically tested.Improvements in software (e.g., control algorithms for robotic arms) mayalso require physical embodiments. Such cyclical prototyping and testingis generally cumulatively expensive and time-consuming.

SUMMARY

Generally, a virtual reality system for providing a virtual roboticsurgical environment may include a virtual reality processor (e.g., aprocessor in a computer implementing instructions stored in memory) forgenerating a virtual robotic surgical environment, a head-mounteddisplay wearable by a user, and one or more handheld controllersmanipulable by the user for interacting with the virtual roboticsurgical environment. The virtual reality processor may, in somevariations, be configured to generate a virtual robotic surgicalenvironment based on at least one predetermined configuration filedescribing a virtual component (e.g., virtual robotic component) in thevirtual environment. The head-mounted display may include an immersivedisplay for displaying the virtual robotic surgical environment to theuser (e.g., with a first-person perspective view of the virtualenvironment). In some variations, the virtual reality system mayadditionally or alternatively include an external display for displayingthe virtual robotic surgical environment. The immersive display and theexternal display, if both are present, may be synchronized to show thesame or similar content. The virtual reality system may be configured togenerate a virtual robotic surgical environment within which a user maynavigate around a virtual operating room and interact with virtualobjects via the head-mounted display and/or handheld controllers. Thevirtual reality system (and variations thereof, as further describedherein) may serve as a useful tool with respect to robotic surgery, inapplications including but not limited to training, simulation, and/orcollaboration among multiple persons.

In some variations, a virtual reality system may interface with a realor actual (non-virtual) operating room. The virtual reality system mayenable visualization of a robotic surgical environment, and may includea virtual reality processor configured to generate a virtual roboticsurgical environment comprising at least one virtual robotic component,and at least one sensor in a robotic surgical environment. The sensormay be in communication with the virtual reality processor andconfigured to detect a status of a robotic component corresponding tothe virtual robotic component. The virtual reality processor isconfigured to receive the detected status of the robotic component andmodify the virtual robotic component based at least in part on thedetected status such that the virtual robotic component mimics therobotic component.

For example, a user may monitor an actual robotic surgical procedure ina real operating room via a virtual reality system that interfaces withthe real operating room (e.g., the user may interact with a virtualreality environment that is reflective of the conditions in the realoperating room). Detected positions of robotic components during asurgical procedure may be compared with their expected positions asdetermined from surgical pre-planning in a virtual environment, suchthat deviations from the surgical plan may trigger a surgeon to performadjustments to avoid collisions (e.g., change a pose of a robotic arm,etc.).

In some variations, the one or more sensors may be configured to detectcharacteristics or status of a robotic component such as position,orientation, speed, and/or velocity. As an illustrative example, the oneor more sensors in the robotic surgical environment may be configured todetect position and/or orientation of a robotic component such as arobotic arm. The position and orientation of the robotic arm may be fedto the virtual reality processor, which moves or otherwise modifies avirtual robotic arm corresponding to the actual robotic arm. As such, auser viewing the virtual robotic surgical environment may visualize theadjusted virtual robotic arm. As another illustrative example, one ormore sensors may be configured to detect a collision involving therobotic component in the robotic surgical environment, and the systemmay provide an alarm notifying the user of the occurrence of thecollision.

Within the virtual reality system, various user modes enable differentkinds of interactions between a user and the virtual robotic surgicalenvironment. For example, one variation of a method for facilitatingnavigation of a virtual robotic surgical environment includes displayinga first-person perspective view of the virtual robotic surgicalenvironment from a first vantage point within the virtual roboticsurgical environment, displaying a first window view of the virtualrobotic surgical environment from a second vantage point and displayinga second window view of the virtual robotic surgical environment from athird vantage point. The first and second window views may be displayedin respective regions of the displayed first-person perspective view.Additionally, the method may include, in response to a user inputassociating the first and second window views, sequentially linking thefirst and second window views to generate a trajectory between thesecond and third vantage points. Window views of the virtual roboticsurgical environment may be displayed at different scale factors (e.g.,“zoom” levels), and may offer views of the virtual environment from anysuitable vantage point in the virtual environment, such as inside avirtual patient, overhead the virtual patient, etc.

In response to a user input indicating selection of a particular windowview, the method may include displaying a new first-person perspectiveview of the virtual environment from the vantage point of the selectedwindow view. In other words, the window views may, for example, operateas portals facilitating transportation between different vantage pointswithin the virtual environment.

As another example of user interaction between a user and the virtualrobotic surgical environment, one variation of a method for facilitatingvisualization of a virtual robotic surgical environment includesdisplaying a first-person perspective view of the virtual roboticsurgical environment from a first vantage point within the virtualrobotic surgical environment, receiving a user input indicatingplacement of a virtual camera at a second vantage point within thevirtual robotic surgical environment different from the first vantagepoint, generating a virtual camera perspective view of the virtualrobotic surgical environment from the second vantage point, anddisplaying the virtual camera perspective view in a region of thedisplayed first-person perspective view. The camera view may, forexample, provide a supplemental view of the virtual environment to theuser that enables the user to monitor various aspects of the environmentsimultaneously while still maintaining primary focus on a main,first-person perspective view. In some variations, the method mayfurther include receiving a user input indicating a selection of avirtual camera type (e.g., movie camera configured to be placed outsidea virtual patient, an endoscopic camera configured to be placed inside avirtual patient, a 360-degree camera, etc.) and displaying a virtualmodel of the selected virtual camera type at the second vantage pointwithin the virtual robotic surgical environment. Other examples of userinteractions with the virtual environment are described herein.

In another variation of a virtual reality system, the virtual realitysystem may simulate a robotic surgical environment in which a user mayoperate both a robotically-controlled surgical instrument using ahandheld controller and a manual laparoscopic surgical instrument (e.g.,while adjacent a patient table, or “over the bed”). For example, avirtual reality system for simulating a robotic surgical environment mayinclude a virtual reality controller configured to generate a virtualrobotic surgical environment comprising at least one virtual robotic armand at least one virtual manual laparoscopic tool, a first handhelddevice communicatively coupled to the virtual reality controller formanipulating the at least one virtual robotic arm in the virtual roboticsurgical environment, and a second handheld device comprising a handheldportion and a tool feature representative of at least a portion of amanual laparoscopic tool, wherein the second handheld device iscommunicatively coupled to the virtual reality controller formanipulating the at least one virtual manual laparoscopic tool in thevirtual robotic surgical environment. For example, in some variations,the tool feature may include a tool shaft and a shaft adapter forcoupling the tool shaft to the handheld portion of the second handhelddevice (e.g., the shaft adapter may include fasteners). The secondhandheld device may be a functioning manual laparoscopic tool or amock-up (e.g., facsimile or genericized version) of a manuallaparoscopic tool, whose movements (e.g., in the tool feature) may bemapped by the virtual reality controller to correspond to movements ofthe virtual manual laparoscopic tool.

The second handheld device may be modular. For example, the tool featuremay be removable from the handheld portion of the second handhelddevice, thereby enabling the second handheld device to function as alaparoscopic handheld device (for controlling a virtual manuallaparoscopic tool) when the tool feature is attached to the handheldportion, as well as a non-laparoscopic handheld device (e.g., forcontrolling a robotically-c ontrolled tool or robotic arm) when the toolfeature is detached from the handheld portion. In some variations, thehandheld portion of the second handheld device may be substantiallysimilar to the first handheld device.

The handheld portion of the second handheld device may include aninteractive feature, such as a trigger or button, which actuates afunction of the virtual manual laparoscopic tool in response toengagement of the interactive feature by a user. For example, a triggeron the handheld portion of the second handheld device may be mapped to avirtual trigger on the virtual manual laparoscopic tool. As anillustrative example, in a variation in which the virtual manuallaparoscopic tool is a virtual manual laparoscopic stapler, a trigger onthe handheld portion may be mapped to firing a virtual staple in thevirtual environment. Other aspects of the system may further approximatethe virtual tool setup in the virtual environment. For example, thevirtual reality system may further include a patient simulator (e.g.,mock patient abdomen) including a cannula configured to receive at leasta portion of the tool feature of the second handheld device, to therebyfurther simulate the user feel of a manual laparoscopic tool.

Generally, a computer-implemented method for operating a virtual roboticsurgical environment may include generating a virtual robotic surgicalenvironment using a client application, where the virtual roboticsurgical environment includes at least one virtual robotic component,and passing information between two software applications in order toeffect movements of the virtual robotic component. For example, inresponse to a user input to move the at least one virtual roboticcomponent in the virtual robotic surgical environment, the method mayinclude passing status information regarding the at least one virtualrobotic component from the client application to a server application,generating an actuation command based on the user input and the statusinformation using the server application, passing the actuation commandfrom the server application to the client application, and moving the atleast one virtual robotic component based on the actuation command. Theclient application and the server application may be run on a sharedprocessor device, or on separate processor devices.

In some variations, passing status information and/or passing theactuation command may include invoking an application programminginterface (API) to support communication between the client and serverapplications. The API may include one or more definitions of datastructures for virtual robotic components and other virtual componentsin the virtual environment. For example, the API may include a pluralityof data structures for a virtual robotic arm, a virtual robotic armsegment (e.g., link), a virtual patient table, a virtual cannula, and/ora virtual surgical instrument. As another example, the API may include adata structure for a virtual touchpoint for allowing manipulation of atleast one virtual robotic component (e.g., virtual robotic arm) or othervirtual component.

For example, the method may include passing status information regardinga virtual robotic arm, such as position and orientation (e.g., pose ofthe virtual robotic arm). The client application may pass such statusinformation to the server application, whereupon the server applicationmay generate an actuation command based on kinematics associated withthe virtual robotic arm.

As described herein, there are various applications and uses for thevirtual reality system. In one variation, the virtual reality system maybe used to expedite the R&D cycle during development of a roboticsurgical system, such as by allowing simulation of potential designwithout the time and significant expense of physical prototypes. Forexample, a method for designing a robotic surgical system may includegenerating a virtual model of a robotic surgical system, testing thevirtual model of the robotic surgical system in a virtual operating roomenvironment, modifying the virtual model of the robotic surgical systembased on the testing, and generating a real model of the roboticsurgical system based on the modified virtual model. Testing the virtualmodel may, for example, involve performing a virtual surgical procedureusing a virtual robotic arm and a virtual surgical instrument supportedby the virtual robotic arm, such as through the client applicationdescribed herein. During a test, the system may detect one or morecollision events involving the virtual robotic arm, which may, forexample, trigger a modification to the virtual model (e.g., modifyingthe virtual robotic arm in link length, diameter, etc.) in response tothe detected collision event. Further testing of the modified virtualmodel may then be performed, to thereby confirm whether the modificationreduced the likelihood of the collision event occurring during thevirtual surgical procedure. Accordingly, testing and modifying roboticsurgical system designs in a virtual environment may be used to identifyissues before testing physical prototypes of the designs.

In another variation, the virtual reality system may be used to test acontrol mode for a robotic surgical component. For example, a method fortesting a control mode for a robotic surgical component may includegenerating a virtual robotic surgical environment, the virtual roboticsurgical environment comprising at least one virtual robotic componentcorresponding to the robotic surgical component, emulating a controlmode for the robotic surgical component in the virtual robotic surgicalenvironment, and, in response to a user input to move the at least onevirtual robotic component, moving the at least one virtual roboticcomponent in accordance with the emulated control mode. In somevariations, moving the virtual robotic component may include passingstatus information regarding the at least one virtual robotic componentfrom a first application (e.g., virtual operating environmentapplication) to a second application (e.g., kinematics application),generating an actuation command based on the status information and theemulated control mode, passing the actuation command from the secondapplication to the first application, and moving the at least onevirtual robotic component in the virtual robotic surgical environmentbased on the actuation command.

For example, the control mode to be tested may be a trajectory followingcontrol mode for a robotic arm. In trajectory following, movement of therobotic arm may be programmed then emulated using the virtual realitysystem. Accordingly, when the system is used to emulate a trajectoryfollowing control mode, the actuation command generated by a kinematicsapplication may include generating an actuated command for each of aplurality of virtual joints in the virtual robotic arm. This set ofactuated commands may be implemented by a virtual operating environmentapplication to move the virtual robotic arm in the virtual environment,thereby allowing testing for collision, volume or workspace of movement,etc.

Other variations and examples of virtual reality systems, their usermodes and interactions, and applications and uses of the virtual realitysystems, are described in further detail herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts an example of an operating room arrangement with arobotic surgical system and a surgeon console. FIG. 1B is a schematicillustration of one exemplary variation of a robotic arm manipulator,tool driver, and cannula with a surgical tool.

FIG. 2A is a schematic illustration of one variation of a virtualreality system. FIG. 2B is a schematic illustration of an immersivedisplay for displaying an immersive view of a virtual realityenvironment.

FIG. 3 is a schematic illustration of components of a virtual realitysystem.

FIG. 4A is an exemplary structure for a communication between a virtualreality environment application and a kinematics application for use ina virtual reality system. FIGS. 4B and 4C are tables summarizingexemplary data structures and fields for an application programinterface for communication between the virtual reality environmentapplication and the kinematics application.

FIG. 5A is a schematic illustration of another variation of a virtualreality system including an exemplary variation of a laparoscopichandheld controller. FIG. 5B is a schematic illustration of an immersivedisplay for displaying an immersive view of a virtual realityenvironment including a virtual manual laparoscopic tool controlled bythe laparoscopic handheld controller.

FIG. 6A is a perspective view of an exemplary variation of alaparoscopic handheld controller. FIG. 6B is a schematic illustration ofa virtual manual laparoscopic tool overlaid on part of the laparoscopichandheld controller shown in FIG. 6A. FIGS. 6C-6E are a side view, adetailed partial perspective view, and a partial cross-sectional view,respectively, of the laparoscopic handheld controller shown in FIG. 6A.

FIG. 7 is a schematic illustration of another variation of a virtualreality system interfacing with a robotic surgical environment.

FIG. 8 is a schematic illustration of a displayed menu for selecting oneor more user modes of one variation of a virtual reality system.

FIGS. 9A-9C are schematic illustrations of a virtual robotic surgicalenvironment with exemplary portals.

FIGS. 10A and 10B are schematic illustrations of an exemplary virtualrobotic surgical environment viewed in a flight mode. FIG. 10C is aschematic illustration of a transition region for modifying the view ofthe exemplary virtual robotic surgical environment in flight mode.

FIG. 11 is a schematic illustration of a virtual robotic surgicalenvironment viewed from a vantage point providing an exemplary dollhouseview of a virtual operating room.

FIG. 12 is a schematic illustration of a view of a virtual roboticsurgical environment with an exemplary heads-up display for displayingsupplemental views.

FIG. 13 is a schematic illustration of a display provided by onevariation of a virtual reality system operating in a virtual commandstation mode.

FIG. 14 is a flowchart of an exemplary variation of a method foroperating a user mode menu for selection of user modes in a virtualreality system.

FIG. 15 is a flowchart of an exemplary variation of a method foroperating in an environment view rotation mode in a virtual realitysystem.

FIG. 16 is a flowchart of an exemplary variation of a method foroperating a user mode enabling snap points in a virtual environment.

DETAILED DESCRIPTION

Examples of various aspects and variations of the invention aredescribed herein and illustrated in the accompanying drawings. Thefollowing description is not intended to limit the invention to theseembodiments, but rather to enable a person skilled in the art to makeand use this invention.

Robotic Surgical System Overview

An exemplary robotic surgical system and surgical environment isillustrated in FIG. 1A. As shown in FIG. 1A, a robotic surgical system150 may include one or more robotic arms 160 located at a surgicalplatform (e.g., table, bed, etc.), where end effectors or surgical toolsare attached to the distal ends of the robotic arms 160 for executing asurgical procedure. For example, a robotic surgical system 150 mayinclude, as shown in the exemplary schematic of FIG. 1B, at least onerobotic arm 160 coupled to a surgical platform, and a tool driver 170generally attached to a distal end of the robotic arm 160. A cannula 100coupled to the end of the tool driver 170 may receive and guide asurgical instrument 190 (e.g., end effector, camera, etc.). Furthermore,the robotic arm 160 may include a plurality of links that are actuatedso as to position and orient the tool driver 170, which actuates thesurgical instrument 190. The robotic surgical system may further includea control tower 152 (e.g., including a power supply, computingequipment, etc.) and/or other suitable equipment for supportingfunctionality of the robotic components.

In some variations, a user (such as a surgeon or other operator) may usea user console 100 to remotely manipulate the robotic arms 160 and/orsurgical instruments (e.g., tele-operation). The user console 100 may belocated in the same procedure room as the robotic system 150, as shownin FIG. 1A. In other embodiments, the user console 100 may be located inan adjacent or nearby room, or tele-operated from a remote location in adifferent building, city, or country. In one example, the user console100 comprises a seat 110, foot-operated controls 120, one or morehandheld user interface devices 122, and at least one user display 130configured to display, for example, a view of the surgical site inside apatient. For example, as shown in the exemplary user console shown inFIG. 1C, a user located in the seat 110 and viewing the user display 130may manipulate the foot-operated controls 120 and/or handheld userinterface devices to remotely control the robotic arms 160 and/orsurgical instruments.

In some variations, a user may operate the robotic surgical system 150in an “over the bed” (OTB) mode, in which the user is at the patient'sside and simultaneously manipulating a robotically-driven tooldriver/end effector attached thereto (e.g., with a handheld userinterface device 122 held in one hand) and a manual laparoscopic tool.For example, the user's left hand may be manipulating a handheld userinterface device 122 to control a robotic surgical component, while theuser's right hand may be manipulating a manual laparoscopic tool. Thus,in these variations, the user may perform both robotic-assisted MIS andmanual laparoscopic techniques on a patient.

During an exemplary procedure or surgery, the patient is prepped anddraped in a sterile fashion, and anesthesia is achieved. Initial accessto the surgical site may be performed manually with the robotic system150 in a stowed configuration or withdrawn configuration to facilitateaccess to the surgical site. Once access is completed, initialpositioning and/or preparation of the robotic system may be performed.During the surgical procedure, a surgeon or other user in the userconsole 100 may utilize the foot-operated controls 120 and/or userinterface devices 122 to manipulate various end effectors and/or imagingsystems to perform the procedure. Manual assistance may also be providedat the procedure table by sterile-gowned personnel, who may performtasks including but not limited to retracting organs, or performingmanual repositioning or tool exchange involving one or more robotic arms160. Non-sterile personnel may also be present to assist the surgeon atthe user console 100. When the procedure or surgery is completed, therobotic system 150 and/or user console 100 may be configured or set in astate to facilitate one or more post-operative procedures, including butnot limited to robotic system 150 cleaning and/or sterilization, and/orhealthcare record entry or printout, whether electronic or hard copy,such as via the user console 100.

In FIG. 1A, the robotic arms 160 are shown with a table-mounted system,but in other embodiments, the robotic arms may be mounted in a cart,ceiling or sidewall, or other suitable support surface. Thecommunication between the robotic system 150, the user console 100, andany other displays may be via wired and/or wireless connection(s). Anywired connections may be optionally built into the floor and/or walls orceiling. The communication between the user console 100 and the roboticsystem 150 may be wired and/or wireless, and may be proprietary and/orperformed using any of a variety of data communication protocols. Instill other variations, the user console 100 does not include anintegrated display 130, but may provide a video output that can beconnected to output to one or more generic displays, including remotedisplays accessible via the internet or network. The video output orfeed may also be encrypted to ensure privacy and all or portions of thevideo output may be saved to a server or electronic healthcare recordsystem.

In other examples, additional user consoles 100 may be provided, forexample to control additional surgical instruments, and/or to takecontrol of one or more surgical instruments at a primary user console.This will permit, for example, a surgeon to take over or illustrate atechnique during a surgical procedure with medical students andphysicians-in-training, or to assist during complex surgeries requiringmultiple surgeons acting simultaneously or in a coordinated manner.

Virtual Reality System

A virtual reality system for providing a virtual robotic surgicalenvironment is described herein. As shown in FIG. 2A, a virtual realitysystem 200 may include a virtual reality processor 210 (e.g., aprocessor in a computer implementing instructions stored in memory) forgenerating a virtual robotic surgical environment, a head-mounteddisplay 220 wearable by a user U, and one or more handheld controllers230 manipulable by the user U for interacting with the virtual roboticsurgical environment. As shown in FIG. 2B, the head-mounted display 220may include an immersive display 222 for displaying the virtual roboticsurgical environment to the user U (e.g., with a first-personperspective view of the virtual environment). The immersive display may,for example, be a stereoscopic display provided by eyepiece assemblies.In some variations, the virtual reality system 200 may additionally oralternatively include an external display 240 for displaying the virtualrobotic surgical environment. The immersive display 222 and the externaldisplay 240, if both are present, may be synchronized to show the sameor similar content.

As described in further detail herein, the virtual reality system (andvariations thereof, as further described herein) may serve as a usefultool with respect to robotic surgery, in applications including but notlimited to training, simulation, and/or collaboration among multiplepersons. More specific examples of applications and uses of the virtualreality system are described herein.

Generally, the virtual reality processor is configured to generate avirtual robotic surgical environment within which a user may navigatearound a virtual operating room and interact with virtual objects viathe head-mounted display and/or handheld controllers. For example, avirtual robotic surgical system may be integrated into a virtualoperating room, with one or more virtual robotic components havingthree-dimensional meshes and selected characteristics (e.g., dimensionsand kinematic constraints of virtual robotic arms and/or virtualsurgical tools, number and arrangement thereof, etc.). Other virtualobjects, such as a virtual control towers or other virtual equipmentrepresenting equipment supporting the robotic surgical system, a virtualpatient, a virtual table or other surface for the patient, virtualmedical staff, a virtual user console, etc., may also be integrated intothe virtual reality operating room.

In some variations, the head-mounted display 220 and/or the handheldcontrollers 230 may be modified versions of those included in anysuitable virtual reality hardware system that is commercially availablefor applications including virtual and augmented reality environments(e.g., for gaming and/or military purposes) and are familiar to one ofordinary skill in the art. For example, the head-mounted display 220and/or the handheld controllers 230 may be modified to enableinteraction by a user with a virtual robotic surgical environment (e.g.,a handheld controller 230 may be modified as described below to operateas a laparoscopic handheld controller). The handheld controller mayinclude, for example, a carried device (e.g., wand, remote device, etc.)and/or a garment worn on the user's hand (e.g., gloves, rings,wristbands, etc.) and including sensors and/or configured to cooperatewith external sensors to thereby provide tracking of the user's hand(s),individual finger(s), wrist(s), etc. Other suitable controllers mayadditionally or alternatively be used (e.g., sleeves configured toprovide tracking of the user's arm(s)).

Generally, a user U may don the head-mounted display 220 and carry (orwear) at least one handheld controller 230 while he or she moves arounda physical workspace, such as a training room. While wearing thehead-mounted display 220, the user may view an immersive first-personperspective view of the virtual robotic surgical environment generatedby the virtual reality processor 210 and displayed onto the immersivedisplay 222. As shown in FIG. 2B, the view displayed onto the immersivedisplay 222 may include one or more graphical representations 230′ ofthe handheld controllers (e.g., virtual models of the handheldcontrollers, virtual models of human hands in place of handheldcontrollers or holding handheld controllers, etc.). A similarfirst-person perspective view may be displayed onto an external display240 (e.g., for assistants, mentors, or other suitable persons to view).As the user moves and navigates within the workspace, the virtualreality processor 210 may change the view of the virtual roboticsurgical environment displayed on the immersive display 222 based atleast in part on the location and orientation of the head-mounteddisplay (and hence the user's location and orientation), therebyallowing the user to feel as if he or she is exploring and moving withinthe virtual robotic surgical environment.

Additionally, the user may further interact with the virtual roboticsurgical environment by moving and/or manipulating the handheldcontrollers 230. For example, the handheld controllers 230 may includeone or more buttons, triggers, touch-sensitive features, scroll wheels,switches, and/or other suitable interactive features that the user maymanipulate to interact with the virtual environment. As the user movesthe handheld controllers 230, the virtual reality processor 210 may movethe graphical representations 230′ of the handheld controllers (or acursor or other representative icon) within the virtual robotic surgicalenvironment. Furthermore, engaging one or more interactive features ofthe handheld controllers 230 may enable the user to manipulate aspectsof the virtual environment. For example, the user may move a handheldcontroller 230 until the graphical representation 230′ of the handheldcontroller is in proximity to a virtual touchpoint (e.g., selectablelocation) on a virtual robotic arm in the environment, engage a triggeror other interactive feature on the handheld controller 230 to selectthe virtual touchpoint, then move the handheld controller 230 whileengaging the trigger to drag or otherwise manipulate the virtual roboticarm via the virtual touchpoint. Other examples of user interactions withthe virtual robotic surgical environment are described in further detailbelow.

In some variations, the virtual reality system may engage other sensesof the user. For example, the virtual reality system may include one ormore audio devices (e.g., headphones for the user, speakers, etc.) forrelaying audio feedback to the user. As another example, the virtualreality system may provide tactile feedback, such as vibration, in oneor more of the handheld controllers 230, the head-mounted display 220,or other haptic devices contacting the user (e.g., gloves, wristbands,etc.).

Virtual Reality Processor

The virtual reality processor 210 may be configured to generate avirtual robotic surgical environment within which a user may navigatearound a virtual operating room and interact with virtual objects. Ageneral schematic illustrating an exemplary interaction between thevirtual reality processor and at least some components of the virtualreality system is shown in FIG. 3.

In some variations, the virtual reality processor 210 may be incommunication with hardware components such as the head-mounted display220, and/or handheld controllers 230. For example, the virtual realityprocessor 210 may receive input from sensors in the head-mounted display220 to determine location and orientation of the user within thephysical workspace, which may be used to generate a suitable,corresponding first-person perspective view of the virtual environmentto display in the head-mounted display 220 to the user. As anotherexample, the virtual reality control 210 may receive input from sensorsin the handheld controllers 230 to determine location and orientation ofthe handheld controllers 230, which may be used to generate suitablegraphical representations of the handheld controllers 230 to display inthe head-mounted display 220 to the user, as well as translate userinput (for interacting with the virtual environment) into correspondingmodifications of the virtual robotic surgical environment. The virtualreality processor 210 may be coupled to an external display 240 (e.g., amonitor screen) that is visible to the user in a non-immersive mannerand/or to other persons such as assistants or mentors who may wish toview the user's interactions with the virtual environment.

In some variations, the virtual reality processor 210 (or multipleprocessor machines) may be configured to execute one or more softwareapplications for generating the virtual robotic surgical environment.For example, as shown in FIG. 4, the virtual reality processor 210 mayutilize at least two software applications, including a virtualoperating environment application 410 and a kinematics application 420.The virtual operating environment application and the kinematicsapplication may communicate via a client-server model. For example, thevirtual operating environment application may operate as a client, whilethe kinematics application may operate as a server. The virtualoperation environment application 410 and the kinematics application 420may be executed on the same processing machine, or on separateprocessing machines coupled via a computer network (e.g., the client orthe server may be a remote device, or the machines may be on a localcomputer network). Additionally, it should be understood that in othervariations, the virtual operating environment application 410 and/or thekinematics application 420 may interface with other software components.In some variations, the virtual operating environment application 410and the kinematics application 520 may invoke one or more applicationprogram interfaces (APIs), which define the manner in which theapplications communicate with one another.

The virtual operating environment 410 may allow for a description ordefinition of the virtual operating room environment (e.g., theoperating room, operating table, control tower or other components, userconsole, robotic arms, table adapter links coupling robotic arms to theoperating table, etc.). At least some descriptions of the virtualoperating room environment may be saved (e.g., in a model virtualreality component database 202) and provided to the processor asconfiguration files. For example, in some variations, as shown in FIG.3, the virtual reality processor (such as through the virtual operatingenvironment application 410 described above) may be in communicationwith a model virtual reality component database 202 (e.g., stored on aserver, local or remote hard drive, or other suitable memory). The modelvirtual reality component database 202 may store one or moreconfiguration files describing virtual components of the virtual roboticsurgical environment. For example, the database 202 may store filesdescribing different kinds of operating rooms (e.g., varying in roomshape or room dimensions), operating tables or other surfaces on which apatient lies (e.g., varying in size, height, surfaces, materialconstruction, etc.), control towers (e.g., varying in size and shape),user console (e.g., varying in user seat design), robotic arms (e.g.,design of arm links and joints, number and arrangement thereof, numberand location of virtual touchpoints on the arm, etc.), table adapterlinks coupling robotic arms to an operating table (e.g., design of tableadapter links and joints, number and arrangement thereof, etc.), patienttypes (e.g., varying in sex, age, weight, height, girth, etc.) and/ormedical personnel (e.g., generic graphical representations of people,graphical representations of actual medical staff, etc.). As onespecific example, a configuration file in Unified Robot DescriptionFormat (URDF) may store a configuration of a particular robotic arm,including definitions or values for fields such as number of arm links,number of arm joints connecting the arm links, length of each arm link,diameter or girth of each arm link, mass of each arm link, type of armjoint (e.g., roll, pitch, yaw etc.), etc. Additionally, kinematicconstraints may be loaded as a “wrapper” over a virtual roboticcomponent (e.g., arm) to further define the kinematic behavior of thevirtual robotic component. In other variations, the virtual realityprocessor 210 may receive any suitable descriptions of virtualcomponents to load and generate in the virtual robotic surgicalenvironment. Accordingly, the virtual reality processor 210 may receiveand utilize different combinations of configuration files and/or otherdescriptions of virtual components to generate particular virtualrobotic surgical environments.

In some variations, as shown in FIG. 3, the virtual reality processor210 may additionally or alternatively be in communication with a patientrecords database 204, which may store patient-specific information. Suchpatient-specific information may include, for example, patient imagingdata (e.g., X-ray, Mill, CT, ultrasound, etc.), medical histories,and/or patient metrics (e.g., age, weight, height, etc.), though othersuitable patient-specific information may additionally or alternativelybe stored in the patient records database 204. When generating thevirtual robotic surgical environment, the virtual reality processor 210may receive patient-specific information from the patient recordsdatabase 204 and integrate at least some of the received informationinto the virtual reality environment. For example, a realisticrepresentation of the patient's body or other tissue may be generatedand incorporated into the virtual reality environment (e.g., a 3D modelgenerated from a combined stack of 2D images, such as Mill images),which may be useful, for example, for determining desirable arrangementof robotic arms around the patient, optimal port placement, etc.specific to a particular patient, as further described herein. Asanother example, patient imaging data may be overlaid over a portion ofthe user's field of view of the virtual environment (e.g., overlaying anultrasound image of a patient's tissue over the virtual patient'stissue).

In some variations, the virtual reality processor 210 may embed one ormore kinematics algorithms via the kinematics application 420 to atleast partially describe behavior of one or more components of thevirtual robotic system in the virtual robotic surgical environment. Forexample, one or more algorithms may define how a virtual robotic armresponds to user interactions (e.g., moving the virtual robotic arm byselection and manipulation of a touchpoint on the virtual robotic arm),or how a virtual robotic arm operates in a selected control mode. Otherkinematics algorithms, such as those defining operation of a virtualtool driver, a virtual patient table, or other virtual components, mayadditionally or alternatively be embedded in the virtual environment. Byembedding in the virtual environment one or more kinematics algorithmsthat accurately describe behavior of an actual (real) robotic surgicalsystem, the virtual reality processor 210 may permit the virtual roboticsurgical system to function accurately or realistically compared to aphysical implementation of a corresponding real robotic surgical system.For example, the virtual reality processor 210 may embed at least onecontrol algorithm that represents or corresponds to one or more controlmodes defining movement of a robotic component (e.g., arm) in an actualrobotic surgical system.

For example, the kinematics application 420 may allow for a descriptionor definition of one or more virtual control modes, such as for thevirtual robotic arms or other suitable virtual components in the virtualenvironment. Generally, for example, a control mode for a virtualrobotic arm may correspond to a function block that enables the virtualrobotic arm to perform or carry out a particular task. For example, asshown in FIG. 4, a control system 430 may include multiple virtualcontrol modes 432, 434, 436, etc. governing actuation of at least onejoint in the virtual robotic arm. The virtual control modes 432, 434,436, etc. may include at least one primitive mode (which governs theunderlying behavior for actuation of at least one joint) and/or at leastone user mode (which governs higher level, task-specific behavior andmay utilize one or more primitive modes). In some variations, a user mayactivate a virtual touchpoint surface of a virtual robotic arm or othervirtual object, thereby triggering a particular control mode (e.g., viaa state machine or other controller). In some variations, a user maydirectly select a particular control mode through, for example, a menudisplayed in the first-person perspective view of the virtualenvironment.

Examples of primitive virtual control modes include, but are not limitedto, a joint command mode (which allows a user to directly actuate asingle virtual joint individually, and/or multiple virtual jointscollectively), a gravity compensation mode (in which the virtual roboticarm holds itself in a particular pose, with particular position andorientation of the links and joints, without drifting downward due tosimulated gravity), and trajectory following mode (in which the virtualrobotic arm may move to follow a sequence of one or more Cartesian orother trajectory commands). Examples of user modes that incorporate oneor more primitive control modes include, but are not limited to, anidling mode (in which the virtual robotic arm may rest in a current ordefault pose awaiting further commands), a setup mode (in which thevirtual robotic arm may transition to a default setup pose or apredetermined template pose for a particular type of surgicalprocedure), and a docking mode (in which the robotic arm facilitates theprocess in which the user attaches the robotic arm to a part, such aswith gravity compensation, etc.).

Generally, the virtual operating environment application 410 and thekinematics application 420 may communicate with each other via apredefined communication protocol, such as an application programinterface (APIs) that organizes information (e.g., status or othercharacteristics) of virtual objects and other aspects of the virtualenvironment. For example, the API may include data structures thatspecify how to communicate information about virtual objects such as avirtual robotic arm (in whole and/or on a segment-by-segment basis) avirtual table, a virtual table adapter connecting a virtual arm to thevirtual table, a virtual cannula, a virtual tool, a virtual touchpointfor facilitating user interaction with the virtual environment, userinput system, handheld controller devices, etc. Furthermore, the API mayinclude one or more data structures that specify how to communicateinformation about events in the virtual environment (e.g., a collisionevent between two virtual entities) or other aspects relating to thevirtual environment (e.g., reference frame for displaying the virtualenvironment, control system framework, etc.). Exemplary data structuresand exemplary fields for containing their information are listed anddescribed in FIGS. 4B and 4C, though it should be understood that othervariations of the API may include any suitable types, names, and numbersof data structures and exemplary field structures.

In some variations, as generally illustrated schematically in FIG. 4A,the virtual operating environment application 410 passes statusinformation to the kinematics application 420, and the kinematicsapplication 420 passes commands to the virtual operating environmentapplication 410 via the API, where the commands are generated based onthe status information and subsequently used by the virtual realityprocessor 210 to generate changes in the virtual robotic surgicalenvironment. For example, a method for embedding one or more kinematicsalgorithms in a virtual robotic surgical environment for control of avirtual robotic arm may include passing status information regarding atleast a portion of the virtual robotic arm from the virtual operatingenvironment application 410 to the kinematics application 420,algorithmically determining an actuation command to actuate at least onevirtual joint of the virtual robotic arm, and passing the actuationcommand from the kinematics application 420 to virtual operatingenvironment application 410. The virtual reality processor 210 maysubsequently move the virtual robotic arm in accordance with theactuation command.

As an illustrative example for controlling a virtual robotic arm, agravity compensation control mode for a virtual robotic arm may beinvoked, thereby requiring one or more virtual joint actuation commandsin order to counteract simulated gravity forces on the virtual joints inthe virtual robotic arm. The virtual operating environment application410 may pass to the kinematics application 420 relevant statusinformation regarding the virtual robotic arm (e.g., position of atleast a portion of the virtual robotic arm, position of the virtualpatient table to which the virtual robotic arm is mounted, position of avirtual touchpoint that the user may have manipulated to move thevirtual robotic arm, joint angles between adjacent virtual arm links)and status information (e.g., direction of simulated gravitational forceon the virtual robotic arm). Based on the received status informationfrom the virtual operating environment application 410 and knownkinematic and/or dynamic properties of the virtual robotic arm and/orvirtual tool drive attached to the virtual robotic arm (e.g., known froma configuration file, etc.), the control system 430 may algorithmicallydetermine what actuated force at each virtual joint is required tocompensate for the simulated gravitational force acting on that virtualjoint. For example, the control system 430 may utilize a forwardkinematic algorithm, an inverse algorithm, or any suitable algorithm.Once the actuated force command for each relevant virtual joint of thevirtual robotic arm is determined, the kinematics application 420 maysend the force commands to the virtual operating environment application410. The virtual reality processor subsequently may actuate the virtualjoints of the virtual robotic arm in accordance with the force commands,thereby causing the virtual robotic arm to be visualized as maintainingits current pose despite the simulated gravitational force in thevirtual environment (e.g., instead of falling down or collapsing undersimulated gravitational force).

Another example for controlling a virtual robotic arm is trajectoryfollowing for a robotic arm. In trajectory following, movement of therobotic arm may be programmed then emulated using the virtual realitysystem. Accordingly, when the system is used to emulate a trajectoryplanning control mode, the actuation command generated by a kinematicsapplication may include generating an actuated command for each of aplurality of virtual joints in the virtual robotic arm. This set ofactuated commands may be implemented by a virtual operating environmentapplication to move the virtual robotic arm in the virtual environment,thereby allowing testing for collision, volume or workspace of movement,etc.

Other virtual control algorithms for the virtual robotic arm and/orother virtual components (e.g., virtual table adapter links coupling thevirtual robotic arm to a virtual operating table) may be implemented viasimilar communication between the virtual operating environmentapplication 410 and the kinematics application 420.

Although the virtual reality processor 210 is generally referred toherein as a single processor, it should be understood that in somevariations, multiple processors may be used to perform the processorsdescribed herein. The one or more processors may include, for example, aprocessor of a general purpose computer, a special purpose computer orcontroller, or other programmable data processing apparatus orcomponent, etc. Generally, the one or more processors may be configuredto execute instructions stored on any suitable computer readable media.The computer readable media may include, for example, magnetic media,optical media, magneto-optical media and hardware devices that arespecially configured to store and execute program code, such asapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs), ROM and RAM devices, flash memory, EEPROMs, opticaldevices (e.g., CD or DVD), hard drives, floppy drives, or any suitabledevice. Examples of computer program code include machine code, such asproduced by a compiler, and files containing higher-level code that areexecuted by a computer using an interpreter. For example, one variationmay be implemented using C-++, JAVA, or other suitable object-orientedprogramming language and development tools. As another example, anothervariation may be implemented in hardwired circuitry in place of, or incombination with, machine-executable software instructions.

Head-Mounted Display Andhandheld Controllers

As shown in FIG. 2A, a user U may wear a head-mounted display 220 and/orhold one or more handheld controllers 230. The head-mounted display 220and handheld controllers 230 may generally enable a user to navigateand/or interact with the virtual robotic surgical environment generatedby the virtual reality processor 210. The head-mounted display 220and/or handheld controllers 230 may communicate signals to the virtualreality processor 210 via a wired or wireless connection.

In some variations, the head-mounted display 220 and/or the handheldcontrollers 230 may be modified versions of those included in anysuitable virtual reality hardware system that is commercially availablefor applications including virtual and augmented reality environments.For example, the head-mounted display 220 and/or the handheldcontrollers 230 may be modified to enable interaction by a user with avirtual robotic surgical environment (e.g., a handheld controller 230may be modified as described below to operate as a laparoscopic handheldcontroller). In some variations, the virtual reality system may furtherinclude one or more tracking emitters 212 that emit infrared light intoa workspace for the user U. The tracking emitters 212 may, for example,be mounted on a wall, ceiling, fixture, or other suitable mountingsurface. Sensors may be coupled to outward-facing surfaces of thehead-mounted display 220 and/or handheld controllers 230 for detectingthe emitted infrared light. Based on the location of any sensors thatdetect the emitted light and when such sensors detect the emitted lightafter the light is emitted, the virtual reality processor 220 may beconfigured to determine (e.g., through triangulation) the location andorientation of the head-mounted display 220 and/or handheld controllers230 within the workspace. In other variations, other suitable means(e.g., other sensor technologies such as accelerometers or gyroscopes,other sensor arrangements, etc.) may be used to determine location andorientation of the head-mounted display 220 and handheld controllers230.

In some variations, the head-mounted display 220 may include straps(e.g., with buckles, elastic, snaps, etc.) that facilitate mounting ofthe display 220 to the user's head. For example, the head-mounteddisplay 220 may be structured similar to goggles, a headband or headset,a cap, etc. The head-mounted display 220 may include two eyepieceassemblies providing a stereoscopic immersive display, thoughalternatively may include any suitable display.

The handheld controllers 230 may include interactive features that theuser may manipulate to interact with the virtual robotic surgicalenvironment. For example, the handheld controllers 230 may include oneor more buttons, triggers, touch-sensitive features, scroll wheels,switches, and/or other suitable interactive features. Additionally, thehandheld controllers 230 may have any of various form factors, such as awand, pinchers, generally round shapes (e.g., ball or egg-shaped), etc.In some variations, the graphical representations 230′ displayed on thehead-mounted display 220 and/or external display 240 may generally mimicthe form factor of the actual, real handheld controllers 230. In somevariations, the handheld controller may include a carried device (e.g.,wand, remote device, etc.) and/or a garment worn on the user's hand(e.g., gloves, rings, wristbands, etc.) and including sensors and/orconfigured to cooperate with external sensors to thereby providetracking of the user's hand(s), individual finger(s), wrist(s), etc.Other suitable controllers may additionally or alternatively be used(e.g., sleeves configured to provide tracking of the user's arm(s)).

Laparoscopic Handheld Controller

In some variations, as shown in the schematic of FIG. 5A, the handheldcontroller 230 may further include at least one tool feature 232 that isrepresentative of at least a portion of a manual laparoscopic tool,thereby forming a laparoscopic handheld controller 234 that may be usedto control a virtual manual laparoscopic tool. Generally, for example,the tool feature 232 may function to adapt the handheld controller 230into a controller substantially similar in form (e.g., user feel andtouch) to a manual laparoscopic tool. The laparoscopic handheldcontroller 234 may be communicatively coupled to the virtual realityprocessor 210 for manipulating a virtual manual laparoscopic tool in thevirtual robotic surgical environment, and may help enable the user tofeel as if he or she is using an actual manual laparoscopic tool whileinteracting with the virtual robotic surgical environment. In somevariations the laparoscopic handheld device may be a mock-up (e.g.,facsimile or genericized version) of a manual laparoscopic tool, whilein other variations the laparoscopic handheld device may be afunctioning manual laparoscopic tool. Movements of at least a portion ofthe laparoscopic handheld controller may be mapped by the virtualreality controller to correspond to movements of the virtual manuallaparoscopic tool. Thus, in some variations, the virtual reality systemmay simulate use of a manual laparoscopic tool for manual MIS.

As shown in FIG. 5A, the laparoscopic handheld controller 234 may beused with a mock patient setup to further simulate the feel of a virtualmanual laparoscopic tool. For example, the laparoscopic handheldcontroller 234 may be inserted into a cannula 250 (e.g., an actualcannula used in MIS procedures to provide realistic feel of a manualtool within a cannula, or a suitable representation thereof, such as atube with a lumen for receiving a tool shaft portion of the laparoscopichandheld controller 234). The cannula 250 may be placed in a mockpatient abdomen 260, such as a foam body with one or more insertionsites or ports for receiving the cannula 250. Alternatively, othersuitable mock patient setups may be used, such as a cavity providingresistance (e.g., with fluid, etc.) with similar feel as an actualpatient abdomen.

Additionally, as shown in FIG. 5B, the virtual reality processor maygenerate a virtual robotic surgical environment including a virtualmanual laparoscopic tool 236′ and/or a virtual cannula 250′ relative toa virtual patient (e.g., the graphical representation 250′ of thecannula depicted as inserted in the virtual patient). As such, thevirtual environment with the virtual manual laparoscopic tool 236′ andvirtual cannula 250′ may be displayed on the immersive display providedby the head-mounted display 220, and/or the external display 240. Acalibration procedure may be performed to map the laparoscopic handheldcontroller 234 to the virtual manual laparoscopic tool 236′ within thevirtual environment. Accordingly, as the user moves and manipulates thelaparoscopic handheld controller 234, the combination of the at leastone tool feature 234 and the mock patient setup may allow the user totactilely feel as if he or she is using a manual laparoscopic tool inthe virtual robotic surgical environment. Likewise, as the user movesand manipulates the laparoscopic handheld controller 234, thecorresponding movements of the virtual manual laparoscopic tool 236′ mayallow the user to visualize the simulation that he or she is using amanual laparoscopic tool in the virtual robotic surgical environment.

In some variations, the calibration procedure for the laparoscopichandheld controller generally maps the laparoscopic handheld controller234 to the virtual manual laparoscopic tool 236′. For example,generally, the calibration procedure may “zero” its position relative toa reference point within the virtual environment. In an exemplarycalibration procedure, the user may insert the laparoscopic handheldcontroller through a cannula 250 into a mock patient abdomen 260, whichmay be placed on a table in front of the user (e.g., at a height that isrepresentative of the height of a real operating patient table). Theuser may continue inserting the laparoscopic handheld controller intothe mock patient abdomen 260 into a suitable depth representative ofdepth achieved during a real laparoscopic procedure. Once thelaparoscopic handheld controller is suitably placed in the mock patientabdomen 260, the user may provide an input (e.g., squeeze a trigger orpush a button on the laparoscopic handheld controller, by voice command,etc.) to confirm and orient the virtual patient to the location andheight of the mock patient abdomen 260. Additionally, other aspects ofthe virtual environment may be calibrated to align with the real,tangible aspects of the system, such as by depicting virtual componentsadjustably movable to target locations and allowing user input toconfirm new alignment of the virtual component with target locations(e.g., by squeezing a trigger or pushing a button on the laparoscopichandheld controller, voice command, etc.). Orientation of virtualcomponents (e.g., rotational orientation of a shaft) may be adjustedwith a touchpad, trackball, or other suitable input on the laparoscopichandheld controller or other device. For example, the virtual operatingroom may be aligned with the real room in which the user is standing, adistal end of the virtual cannula or trocar may be aligned with the realentry location in the mock patient abdomen, etc. Furthermore, in somevariations, a virtual end effector (e.g., endocutter, clipper) may belocated and oriented via the laparoscopic handheld controller to a newtarget location and orientation in similar manners.

In some variations, as shown in FIG. 5B, the system may include both ahandheld controller 230 and a laparoscopic handheld controller 234.Accordingly, the virtual reality processor may generate a virtualenvironment including both a graphical representation 230′ of a handheldcontroller 230 (with no laparoscopic attachment) and a virtual manuallaparoscopic tool 236′ as described above. The handheld controller 230may be communicatively coupled to the virtual reality processor 210 formanipulating at least one virtual robotic arm, and the laparoscopichandheld controller 234 may be communicatively coupled to the virtualreality processor 210 for manipulating a virtual manual laparoscopictool 236′. Thus, in some variations, the virtual reality system maysimulate an “over the bed” mode of using a robotic surgical system, inwhich an operator is at the patient's side and manipulating both arobotic arm (e.g., with one hand) providing robotic-assisted MIS, and amanual laparoscopic tool providing manual MIS.

The tool feature 232 may include any suitable feature generallyapproximating or representing a portion of a manual laparoscopic tool.For example, the tool feature 232 may generally approximate alaparoscopic tool shaft (e.g., include an elongated member extendingfrom a handheld portion of the controller). As another example, the toolfeature 232 may include a trigger, button, or other laparoscopicinteractive feature similar to that present on a manual laparoscopictool that engages an interactive feature on the handheld controller 230but provides a realistic form factor mimicking the feel of a manuallaparoscopic tool (e.g., the tool feature 232 may include a largertrigger having a realistic form factor that is overlaid with and engagesa generic interactive feature on the handheld controller 230). As yetanother example, the tool feature 232 may include selected materialsand/or masses to create a laparoscopic handheld controller 234 having aweight distribution that is similar to a particular kind of manuallaparoscopic tool. In some variations, the tool feature 232 may includeplastic (e.g., polycarbonate, acrylonitrile butadiene styrene (ABS),nylon, etc.) that is injection molded, machined, 3D-printed, or othersuitable material shaped in any suitable fashion. In other variations,the tool feature 232 may include metal or other suitable material thatis machined, casted, etc.

In some variations, the tool feature 236 may be an adapter or otherattachment that is formed separately from the handheld controller 230and coupled to the handheld controller 230 via fasteners (e.g., screws,magnets, etc.), interlocking features (e.g., threads or snap fitfeatures such as tabs and slots, etc.), epoxy, welding (e.g., ultrasonicwelding), etc. The tool feature 236 may be reversibly coupled to thehandheld controller 230. For example, the tool feature 236 may beselectively attached to the handheld controller 230 in order to adapt ahandheld controller 230 when a laparoscopic-style handheld controller230 is desired, while the tool feature 236 may be selectively detachedfrom the handheld controller 230 when a laparoscopic-style handheldcontroller 230 is not desired. Alternatively, the tool feature 236 maybe permanently coupled to the handheld portion 234, such as duringmanufacturing. Furthermore, in some variations, the handheld portion 234and the tool feature 236 may be integrally formed (e.g., injectionmolded together as a single piece).

One exemplary variation of a laparoscopic handheld controller is shownin FIG. 6A. The laparoscopic handheld controller 600 may include ahandheld portion 610 (e.g., similar to handheld controller 230 describedabove), a tool shaft 630, and a shaft adapter 620 for coupling the toolshaft 630 to the handheld portion 610. As shown in FIG. 6B, thelaparoscopic handheld controller 600 may generally be used to control avirtual manual laparoscopic stapler tool 600′, though the laparoscopichandheld controller 600 may be used to control other kinds of virtualmanual laparoscopic tools (e.g., scissors, dissectors, graspers,needleholders, probes, forceps, biopsy tools, etc. For example, thehandheld portion 610 may be associated with a virtual handle 610′ of thevirtual manual laparoscopic stapler tool 600′ having a stapler endeffector 640′, such that the user's manipulation of the handheld portion610 is mapped to manipulation of the virtual handle 610′. Similarly, thetool shaft 630 may correspond to a virtual tool shaft 630′ of thevirtual manual laparoscopic stapler tool 600′. The tool shaft 630 andthe virtual tool shaft 630′ may be inserted into a cannula and a virtualcannula, respectively, such that movement of the tool shaft 630 relativeto the cannula is mapped to movement of the virtual tool shaft 630′within the virtual cannula in the virtual robotic surgical environment.

The handheld portion 610 may include one or more interactive features,such as finger trigger 612 and/or button 614, which may receive userinput from the user's fingers, palms, etc. and be communicativelycoupled to a virtual reality processor. In this exemplary embodiment,the finger trigger 612 may be mapped to a virtual trigger 612′ on thevirtual manual laparoscopic stapler tool 600′. The virtual trigger 612′may be visualized as actuating the virtual end effector 640′ (e.g.,causing the virtual members of the virtual end effector 640′ to closeand fire staplers) for stapling virtual tissue in the virtualenvironment. Accordingly, when the user actuates the finger trigger 612on the laparoscopic handheld controller, the signal from finger trigger612 may be communicated to the virtual reality processor, which modifiesthe virtual manual laparoscopic stapler tool 600′ to interact within thevirtual environment in simulation of an actual manual laparoscopicstapler tool. In another variation, a trigger attachment may physicallyresemble (e.g., in shape and form) the virtual trigger 612′ on thevirtual manual laparoscopic stapler tool 600′ and may be coupled to thefinger trigger 612, which may enable the laparoscopic handheldcontroller 600 to even more closely mimic the user feel of the virtualmanual laparoscopic stapler tool 600′.

As shown in FIGS. 6C-6E, the shaft adapter 620 may generally function tocouple the tool shaft 630 to the handheld portion 610, which may, forexample, adapt a handheld controller (similar to handheld controller 210described above) into a laparoscopic handheld controller. The shaftadapter 620 may generally include a first end for coupling to thehandheld portion 610 and a second end for coupling to the tool shaft630. As shown best in FIG. 6E, the first end of the shaft adapter 620may include a proximal portion 620 a and distal portion 620 b configuredto clamp on a feature of the handheld portion 610. For example, thehandheld portion 610 may include generally ring-like portion defining acentral space 614 which receives the proximal portion 620 a and thedistal portion 620 b. The proximal portion 620 a and the distal portion620 b may clamp on either side of the ring-like portion at its innerdiameter, and be fixed to the ring-like portion via fasteners (notshown) passing through fastener holes 622, thereby securing the shaftadapter 620 to the handheld portion 610. Additionally or alternatively,the shaft adapter 620 may couple to the handheld portion 610 in anysuitable fashion, such as an interference fit, epoxy, interlockingfeatures (e.g., between the proximal portion 620 a and the distalportion 620 b), etc. As also shown in FIG. 6E, the second end of theshaft adapter 620 may include a recess for receiving the tool shaft 620.For example, the recess may be generally cylindrical for receiving agenerally cylindrical end of a tool shaft portion 630, such as through apress fit, friction fit, or other interference fit. Additionally oralternatively, the tool shaft 620 may be coupled to the shaft adapter620 with fasteners (e.g., screws, bolts, epoxy, ultrasonic welding,etc.). The tool shaft 630 may be any suitable size (e.g., length,diameter) for mimicking or representing a manual laparoscopic tool.

In some variations, the shaft adapter 620 may be selectively removablefrom the handheld portion 610 for permitting selective use of thehandheld portion 610 both as a standalone handheld controller (e.g.,handheld controller 210) and as a laparoscopic handheld controller 600.Additionally or alternatively, the tool shaft 630 may be selectivelyremovable from the shaft adapter 620 (e.g., although the shaft adapter620 may be intentionally fixed to the handheld portion 610, the toolshaft 620 may be selectively removable from the shaft adapter 620 toconvert the laparoscopic handheld control 600 to a standalone handheldcontroller 210).

Generally, the tool feature of the laparoscopic handheld controller 600,such as the shaft adapter 620 and the tool shaft 630, may be made of arigid or semi-rigid plastic or metal, and may be formed through anysuitable manufacturing process, such as 3D printing, injection molding,milling, turning, etc. The tool feature may include multiple kinds ofmaterials, and/or weights or other masses to further simulate the userfeel of a particular manual laparoscopic tool.

System Variations

One or more aspects of the virtual reality system described above may beincorporated into other variations of systems. For example, in somevariations, a virtual reality system for providing a virtual roboticsurgical environment may interface with one or more features of a realrobotic surgical environment. For example, as shown in FIG. 3, a system700 may include one or more processors (e.g., a virtual realityprocessor 210) configured to generate a virtual robotic surgicalenvironment, and one or more sensors 750 in a robotic surgicalenvironment, where the one or more sensors 750 is in communication withthe one or more processors. Sensor information from the robotic surgicalenvironment may be configured to detect status of an aspect of therobotic surgical environment, such for mimicking or replicating featuresof the robotic surgical environment in the virtual robotic surgicalenvironment. For example, a user may monitor an actual robotic surgicalprocedure in a real operating room via a virtual reality system thatinterfaces with the real operating room (e.g., the user may interactwith a virtual reality environment that is reflective of the conditionsin the real operating room).

In some variations, one or more sensors 750 may be configured to detectstatus of at least one robotic component (e.g., a component of a roboticsurgical system, such as a robotic arm, a tool driver coupled to arobotic arm, a patient operating table to which a robotic arm isattached, a control tower, etc.) or other component of a roboticsurgical operating room. Such status may indicate, for example,position, orientation, speed, velocity, operative state (e.g., on oroff, power level, mode), or any other suitable status of the component.

For example, one or more accelerometers may be coupled to a robotic armlink and be configured to provide information about the robotic armlink's position, orientation, and/or velocity of movement, etc. Multipleaccelerometers on multiple robotic arms may be configured to provideinformation regarding impending and/or present collisions betweenrobotic arms, between different links of a robotic arm, or between arobotic arm and a nearby obstacle having a known position.

As another example, one or more proximity sensors (e.g., infraredsensor, capacitive sensor) may be coupled to a portion of a robotic armor other components of the robotic surgical system or surgicalenvironment. Such proximity sensors may, for example, be configured toprovide information regarding impending collisions between objects.Additionally or alternatively, contact or touch sensors may be coupledto a portion of a robotic arm or other components of the roboticsurgical environment, and may be configured to provide informationregarding a present collision between objects.

In another example, one or more components of the robotic surgicalsystem or surgical environment may include markers (e.g., infraredmarkers) to facilitate optical tracking of the position, orientation,and/or velocity of various components, such as with overhead sensorsmonitoring the markers in the surgical environment. Similarly, thesurgical environment may additionally or alternatively include camerasfor scanning and/or modeling the surgical environment and its contents.Such optical tracking sensors and/or cameras may be configured toprovide information regarding impending and/or present collisionsbetween objects.

As another example, one or more sensors 750 may be configured to detecta status of a patient, a surgeon, or other surgical staff. Such statusmay indicate, for example, position, orientation, speed, velocity,and/or biological metrics such as heart rate, blood pressure,temperature, etc. For example, a heart rate monitor, a blood pressuremonitor, thermometer, and/or oxygenation sensor, etc. may be coupled tothe patient and enable a user to keep track of the patient's condition.

Generally, in these variations, a virtual reality processor 210 maygenerate a virtual robotic surgical environment similar to thatdescribed elsewhere herein. Additionally, upon receiving statusinformation from the one or more sensors 750, the virtual realityprocessor 210 or other processor in the system may incorporate thedetected status in any one or more suitable ways. For example, in onevariation, the virtual reality processor 210 may be configured togenerate a virtual reality replica or near-replica of a robotic surgicalenvironment and/or a robotic surgical procedure performed therein. Forexample, the one or more sensors 750 in the robotic surgical environmentmay be configured to detect a status of a robotic componentcorresponding to a virtual robotic component in the virtual roboticsurgical environment (e.g., the virtual robotic component may besubstantially representative of the robotic component in visual formand/or function). In this variation, the virtual reality processor 210may be configured to receive the detected status of the roboticcomponent, and then modify the virtual robotic component based at leastin part on the detected status such that the virtual robotic componentmimics the robotic component. For example, if a surgeon moves a roboticarm during a robotic surgical procedure to a particular pose, then avirtual robotic arm in the virtual environment may move correspondingly.

As another example, the virtual reality processor 210 may receive statusinformation indicating an alarm event, such as an impending or presentcollision between objects, or poor patient health condition. Uponreceiving such information, the virtual reality processor 210 mayprovide a warning or alarm to the user of the occurrence of the event,such as by displaying a visual alert (e.g., text, icon indicatingcollision, a view within the virtual environment depicting thecollision, etc.), audio alert, etc.

As yet another example, the one or more sensors in the robotic surgicalenvironment may be used to compare an actual surgical procedure(occurring in the non-virtual robotic surgical environment) with aplanned surgical procedure as planned in a virtual robotic surgicalenvironment. For example, an expected position of at least one roboticcomponent (e.g., robotic arm) may be determined during surgicalpreplanning, as visualized as a corresponding virtual robotic componentin a virtual robotic surgical environment. During an actual surgicalprocedure, one or more sensors may provide information about a measuredposition of the actual robotic component. Any differences between theexpected and measured position of the robotic component may indicatedeviations from a surgical plan that was constructed in the virtualreality environment. Since such deviations may eventually result inundesired consequences (e.g., unintended collisions between roboticarms, etc.), identification of deviations may allow the user to adjustthe surgical plan accordingly (e.g., reconfigure approach to a surgicalsite, change surgical instruments, etc.).

User Modes

Generally, the virtual reality system may include one or more user modesenabling a user to interact with the virtual robotic surgicalenvironment by moving and/or manipulating the handheld controllers 230.Such interactions may include, for example, moving virtual objects(e.g., virtual robotic arm, virtual tool, etc.) in the virtualenvironment, adding camera viewpoints to view the virtual environmentsimultaneously from multiple vantage points, navigate within the virtualenvironment without requiring the user to move the head-mounted display220 (e.g., by walking), etc. as further described below.

In some variations, the virtual reality system may include a pluralityof user modes, where each user mode is associated with a respectivesubset of user interactions. As shown in FIG. 8, at least some of theuser modes may be shown on a display (e.g., head-mounted display 220)for user selection. For example, at least some of the user modes maycorrespond to selectable user mode icons 812 displayed in a user modemenu 810. The user mode menu 810 may be overlaid on the display of thevirtual robotic surgical environment such that a graphicalrepresentation 230′ of the handheld controller (or user hand, othersuitable representative icon, etc.) may be maneuvered by the user toselect a user mode icon, thereby activating the user mode correspondingto the selected user mode icon. As shown in FIG. 8, the user mode icons812 may be generally arranged in a palette or circle, but may bealternatively arranged in a grid or other suitable arrangement. In somevariations, a selected subset of possible user modes may be presented inthe menu 810 based on, for example, user preferences (e.g., associatedwith a set of user login information), preferences of users similar tothe current user, type of surgical procedure, etc.

FIG. 14 illustrates a method of operation 1400 of an exemplary variationof a user mode menu providing selection of one or more user mode icons.To activate the user menu, the user may activate a user input methodassociated with the menu. For example, an input method may be activatedby a user engaging with a handheld controller (e.g., handheld userinterface device), such as by pressing a button or other suitablefeature on the handheld controller (1410). As another example, an inputmethod may be activated by a user engaging a pedal or other feature of auser console (1410′). Voice commands and/or other devices mayadditionally or alternatively be used to activate an input methodassociated with the menu. While the input method is engaged (1412), thevirtual reality system may render and display an array of user modeicons (e.g., arranged in a palette around a central origin as shown inFIG. 8A). The array of user mode icons may be generally displayed nearor around a graphical representation of the handheld controller and/orat a rendered cursor that is controlled by the handheld controller.

For example, in one variation in which a handheld controller includes acircular menu button and a graphical representation of the handheldcontroller also has a circular menu button displayed in the virtualreality environment, the array of user mode icons may be centered aroundand aligned with the menu button such that the normal vectors of themenu plane and menu button are substantially aligned. The circular orradial menu may include, for example, multiple different menu regions(1414) or sectors, each of which may be associated with an angle range(e.g., an arcuate segment of the circular menu) and a user mode icon(e.g., as shown in FIG. 8). Each region may be toggled between aselected state and an unselected state.

The method 1400 may generally include determining selection of a usermode by the user and receiving confirmation that the user would like toactivate the selected user mode for the virtual reality system. Toselect a user mode in the user mode menu, the user may move the handheldcontroller (1420) to freely manipulate the graphical representation ofthe handheld controller and navigate through the user mode icons in theuser mode menu. Generally, the position/orientation of the handheldcontroller (and position/orientation of the graphical representation ofthe handheld controller which moves in correspondence with the handheldcontroller) may be analyzed to determine whether the user has selected aparticular user mode icon. For example, in variations in which the usermode icons are arranged in a generally circular palette around a centralorigin, the method may include determining radial distance and/orangular orientation of the graphical representation of the handheldcontroller relative to the central origin. For example, a test fordetermining user selection of a user mode icon may include one or moreprongs, which may be satisfied in any suitable order. In a first prong(1422), the distance of the graphical representation of the handheldcontroller to the center of the user mode menu (or another referencepoint in the user mode menu) is compared to a distance threshold. Thedistance may be expressed in terms of absolute distance (e.g., number ofpixels) or ratios (e.g., percentage of distance between a center pointand the user mode icons arranged around the periphery of the user modemenu, such as 80% or more). If the distance is less than the threshold,then it may be determined that no user mode icon is selected.Additionally or alternatively, selection of a user mode icon may dependon a second prong (1424). In the second prong (1424), the orientation ofthe graphical representation of the handheld controller is measured andcorrelated to a user mode icon associated with an arcuate segment of themenu. If the orientation is corresponds to a selected arcuate segment ofthe menu, then it may be determined that a particular user mode(associated with the selected arcuate segment) is selected by the user.For example, a user mode icon may be determined as selected by the userif both the distance and the angular orientation of the graphicalrepresentation of the handheld controller relative to the origin satisfythe conditions (1422) and (1424).

After determining that a user has selected a particular user mode icon,the method may, in some variations, convey such selection to the user(e.g., as confirmation) by visual and/or auditory indications. Forexample, in some variations, the method may include rendering one ormore visual cues (1430) in the displayed virtual reality environment inresponse to determining that a user has selected a user mode icon. Asshown in FIG. 14, exemplary visual cues (1432) include modifying theappearance of the selected user mode icon (and/or the arcuate segmentassociated with the selected user mode icon) with highlighting (e.g.,thickened outlines), animation (e.g., wiggling lines, “dancing” or“pulsating” icon), change in size (e.g., enlargement of icon), change inapparent depth, change in color or opacity (e.g., more or lesstranslucent, change in pattern fill of icon), change in position (e.g.,move radially outward or inward from the central origin, etc.), and/orany suitable visual modification. In some variations, indicating to theuser in these or any suitable manner may inform the user which user modewill be activated, prior to the user confirming the selection of aparticular user mode. For example, the method may include rendering oneor more visual cues (1430) as the user navigates or scrolls through thevarious user mode icons in the menu.

The user may confirm approval of the selected user mode icon in one ormore various manners. For example, the user may release or deactivatethe user input method (1440) associated with the menu (e.g., releasing abutton on the handheld controller, disengaging a foot pedal), such as toindicate approval of the selected user mode. In other variations, theuser may confirm selection by hovering over the selected user mode iconfor at least a predetermined period of time (e.g., at least 5 seconds),double-clicking the user input method associated with the user menu(e.g., double-clicking the button, etc.), speaking a verbal commandindicating approval, etc.

In some variations, upon receiving confirmation that the user approvesthe selected user mode, the method may include verifying which user modeicon has been selected. For example, as shown in FIG. 14, a test forverifying which user mode icon has been selected may include one or moreprongs, which may be satisfied in any suitable order. For example, invariations in which the user mode icons are arranged in a generallycircular palette around a central origin, the method may includedetermining radial distance relative to the central origin (1442) and/orangular orientation of the graphical representation of the handheldcontroller relative to the central origin (1446) when the user indicatesapproval of user mode icon selection. In some variations, prongs (1442)and (1446) may be similar to prongs (1422) and (1424) described above,respectively. If at least one of these prongs (1442) and (1444) is notsatisfied, then the release of the user input method may correlated to anon-selection of a user mode icon (e.g., the user may have changed hisor her mind about selecting a new user mode). Accordingly, if thegraphical representation of the handheld controller fails to satisfy thedistance threshold (1442) then the original or previous user mode may beretained (1444). Similarly, if the graphical representation of thehandheld controller fails to correspond to an arcuate segment of themenu (1446), then the original or previous user mode may be retained(1448). If the graphical representation of the handheld controller doessatisfy the distance threshold (1442) and corresponds to an arcuatesegment of the menu, then the selected user mode may be activated(1450). In other variations, a user mode may additionally oralternatively be selected with other interactions, such as voicecommand, eye-tracking via sensors, etc. Furthermore, the system mayadditionally or alternatively suggest activation of one or more usermodes based on criteria such as user activity within the (e.g., if theuser is frequently turning his head to see detail on the edge of hisfield of view, the system may suggest a user mode enabling placement ofa camera to provide a heads-up display window view from a desiredvantage point, as described below), type of surgical procedure, etc.

Object Gripping

One exemplary user mode with the virtual robotic surgical environmentenables a user to grip, move, or otherwise manipulate virtual objects inthe virtual environment. Examples of manipulable virtual objectsinclude, but are not limited to, virtual representations of physicalitems (e.g., one or more virtual robotic arms, one or more virtual tooldrivers, virtual manual laparoscopic tools, virtual patient operatingtable or other resting surface, virtual control tower or otherequipment, virtual user console, etc.) and other virtual or graphicalconstructs such as portals, window display, patient imaging or otherprojections on a heads-up display, etc. which are further describedbelow.

At least some of the virtual objects may include or be associated withat least one virtual touchpoint or selectable feature. When the virtualtouchpoint is selected by a user, the user may move (e.g., adjustposition and/or orientation) the virtual object associated with theselected virtual touchpoint. Furthermore, multiple virtual touchpointsmay be simultaneously selected (e.g., with multiple handheld controllers230 and their graphical representations 230′) on the same virtual objector multiple separate virtual objects.

The user may generally select a virtual touchpoint by moving a handheldcontroller 230 to correspondingly move a graphical representation 230′to the virtual touchpoint in the virtual environment, then engaging aninteractive feature such as a trigger or button on the handheldcontroller 230 to indicate selection of the virtual touchpoint. In somevariations, a virtual touchpoint may remain selected as long as the userengages the interactive feature on the handheld controller 230 (e.g., aslong as the user depresses a trigger) and may become unselected when theuser releases the interactive feature. For example, the virtualtouchpoint may enable the user to “click and drag” the virtual objectvia the virtual touchpoint. In some variations, a virtual touchpoint maybe toggled between a selected state and an unselected state, in that avirtual touchpoint may remain selected after a single engagement of theinteractive feature on the handheld controller until a second engagementof the interactive feature toggles the virtual touchpoint to anunselected state. In the virtual robotic surgical environment, one orboth kinds of virtual touchpoints may be present.

A virtual object may include at least one virtual touchpoint for directmanipulation of the virtual object. For example, a virtual robotic armin the virtual environment may include a virtual touchpoint on one ofits virtual arm links. The user may move a handheld controller 230 untilthe graphical representation 230′ of the handheld controller is inproximity to (e.g., hovering over) the virtual touchpoint, engage atrigger or other interactive feature on the handheld controller 230 toselect the virtual touchpoint, then move the handheld controller 230 tomanipulate the virtual robotic arm via the virtual touchpoint.Accordingly, the user may manipulate the handheld controller 230 inorder to reposition the virtual robotic arm in a new pose, such as tocreate a more spacious workspace in the virtual environment by thepatient, test range of motion of the virtual robotic arm to determinelikelihood of collisions between the virtual robotic arm and otherobjects, etc.

A virtual object may include at least one virtual touchpoint that isassociated with a second virtual object, for indirect manipulation ofthe second virtual object. For example, a virtual control panel mayinclude a virtual touchpoint on a virtual switch or button that isassociated with a patient operating table. The virtual switch or buttonmay, for example, control the height or angle of the virtual patientoperating table in the virtual environment, similar to how a switch orbutton on a real control panel might electronically or mechanicallymodify the height or angle of a real patient operating table. The usermay move a handheld controller 230 until the graphical representation230′ of the handheld controller is in proximity to (e.g., hovering over)the virtual touchpoint, engage a trigger or other interactive feature onthe handheld controller 230 to select the virtual touchpoint, then movethe handheld controller 230 to manipulate the virtual switch or buttonvia the virtual touchpoint. Accordingly, the user may manipulate thehandheld controller 230 in order to modify the height or angle of thevirtual environment, such as to improve angle of approach or access to aworkspace in the virtual environment.

When a virtual touchpoint is selected, the virtual reality processor maymodify the virtual robotic surgical environment to indicate to the userthat the virtual touchpoint is indeed selected. For example, the virtualobject including the virtual touchpoint may be highlighted by beinggraphically rendered in a different color (e.g., blue or red) and/oroutlined in a different line weight or color, in order to visuallycontrast the affected virtual object from other virtual objects in thevirtual environment. Additionally or alternatively, the virtual realityprocessor may provide audio feedback (e.g., a tone, beep, or verbalacknowledgment) through an audio device indicating selection of thevirtual touchpoint, and/or tactile feedback (e.g., a vibration) througha handheld controller 230, the head-mounted display 220, or othersuitable device.

Navigation

Other exemplary user modes with the virtual robotic surgical environmentmay enable a user to navigate and explore the virtual space within thevirtual environment.

Snap Points

In some variations, the system may include a user mode enabling “snappoints”, or virtual targets within a virtual environment which may beused to aid user navigation within the virtual environment. A snap pointmay, for example, be placed at a user-selected or default locationwithin the virtual environment and enable a user to quickly navigate tothat location upon selection of the snap point. A snap point may, insome variations, be associated with an orientation within the virtualenvironment and/or an apparent scale (zoom level) of the display of theenvironment from that vantage point. Snap points may, for example, bevisually indicated as colored dots or other colored markers graphicallydisplayed in the first-person perspective view. By selecting a snappoint, the user may be transported to the vantage point of the selectedsnap point within the virtual robotic surgical environment.

For example, FIG. 16 illustrates method of operation 1600 of anexemplary variation of a user mode enabling snap points. As shown inFIG. 16, a snap point may be positioned (1610) in the virtualenvironment by a user or as a predetermined setting. For example, a usermay navigate through a user mode menu as described above, and select or“grab” a snap point icon from the menu with a handheld controller (e.g.,indicated with a colored dot or other suitable marker) and drag and dropthe snap point icon to a desired location and/or orientation in thevirtual environment. The snap point may, in some variations, berepositioned by the user reselecting the snap point (e.g., moving thegraphical representation of the handheld controller until it intersectswith the snap point or a collision volume boundary around the snappoint, then engaging an input feature such as a button or trigger) anddragging and dropping the snap point icon to a new desired location. Insome variations, the user may set the scale or zoom level of the vantagepoint (1620) associated with the snap point, such by adjusting adisplayed slider bar or scroll wheel, motions as described above forsetting a scale level for an environmental view manipulation, etc. Thesnap point may, in some examples, have a default scale level associatedwith all or a subcategory of snap points, a scale level associated withthe current vantage point of the user when the user places the snappoint, or adjusted as described above. Furthermore, once a snap point isplaced, the snap point may be stored (1630) in memory (e.g., local orremote storage) for future access. A snap point may, in some variations,be deleted from the virtual environment and from memory. For example, asnap point may be selected (in a similar manner as for repositioning ofthe snap point) and designed for deletion by dragging it off-screen to apredetermined location (e.g., virtual trash can) and/or moving it with apredetermined velocity (e.g., “thrown” in a direction away from theuser's vantage point with a speed greater than a predeterminedthreshold), selection of a secondary menu option, voice command, etc.

Once one or more snap points for a virtual environment are stored inmemory, the user may select one of the stored snap points (1640) foruse. For example, upon selection of a stored snap point, the user'svantage point may be adjusted to the position, orientation, and/or scaleof the selected snap point's settings (1650), thereby allowing the userto feel as if they are teleporting to the location associated with theselected snap point. In some variations, the user's previous vantagepoint may be stored as a snap point (1660) to enable easy “undo” of theuser's perceived teleportation and transition the user back to theirprevious vantage point. Such a snap point may be temporary (e.g.,disappear after a predetermined period of time, such as after 5-10seconds). In some examples, the user's previous vantage point may bestored as a snap point only if the user's previous location was not apre-existing snap point. Furthermore, in some variations, a virtualtrail or trajectory (e.g., line or arc) may be displayed in the virtualenvironment connecting the user's previous vantage point to the user'snew vantage point associated with the selected snap point, which may,for example, provide the user with context as to how they haveteleported within the virtual environment. Such a visual indication maybe removed from the display of the virtual environment after apredetermined period of time (e.g., after 5-10 seconds).

Generally, in some variations, a snap point may operate in a similarmanner as portals described below, except that a snap point may indicatea vantage point without providing a window preview of the virtualenvironment. For example, snap points may be placed at user-selectedvantage points outside and/or inside the virtual patient, and may belinked into one or more trajectories, similar to portals as describedabove. In some variations, snap point trajectories may be set by theuser in a manner similar to that described below for portals.

Portals

In some variations, the system may include a user mode that facilitatesplacement of one or more portals, or teleportation points, atuser-selected locations in the virtual environment. Each portal may, forexample, serve as a transportation gateway to a corresponding vantagepoint in the virtual environment, thereby allowing the user to swiftlychange vantage points for viewing and navigating the virtualenvironment. Generally, upon selection (e.g., with one or more handheldcontrollers 230) of a portal, the user's apparent location maytransition to the location of the selected portal, such that the userviews the virtual environment from the selected portal's vantage pointand has the sensation of “jumping” around the virtual environment. Byplacing one or more portals around the virtual environment, the user mayhave the ability to quickly move between various vantage points.Placement, adjustment, storing, and/or navigation of portals around thevirtual environment may be similar to that of snap points describedabove.

For example, as generally described above, the system may display afirst-person perspective view of the virtual robotic surgicalenvironment from a first vantage point within the virtual roboticsurgical environment. The user may navigate through a menu to select auser mode that enables placement of a portal. As shown in FIG. 9A, theuser may manipulate the graphical representation 230′ of the handheldcontroller to position a portal 910 in a selected location in thevirtual environment. For example, the user may engage a feature (e.g.,trigger or button) on the handheld controller while a portalplacement-enabling user mode is activated, such that while the featureis engaged and the user moves the position and/or orientation of thehandheld controller, a portal 910 may appear and be moved within thevirtual environment. One or more portal placement indicators 920 (e.g.,one or more arrows, a line, an arc, etc. connecting the graphicalrepresentation 230′ to a prospective portal location) may aid incommunicating to the user the prospective location of a portal 910, suchas by helping with depth perception. Size of the portal 910 may beadjusted by “grabbing” and stretching or shrinking the sides of theportal 910 via the handheld controllers. When the portal 910 location isconfirmed (e.g., by the user releasing the engaged feature on thehandheld controller, double-clicking, etc.), the user's apparentlocation within the virtual environment may be updated to match thevantage point associated with the portal 910. In some variations, asdescribed below, at least some vantage points within the virtuallocation may be prohibited. These prohibited vantage points may bestored in memory (e.g., local or remote storage). In these variations,if a portal 910 location is confirmed in a prohibited location (e.g.,compared to and matched among a list of prohibited vantage points storedin memory), then the user's apparent location within the virtualenvironment may be retained with no changes. However, if a portal 910location is confirmed as permissible (e.g., compared to and not matchedamong the list of prohibited vantage points), then the user's apparentlocation within the virtual environment may be updated as describedabove.

In some variations, once the user has placed the portal 910 at a desiredvantage point, a window view of the virtual environment from the vantagepoint of the placed portal 910 may be displayed within the portal 910,thereby offering a “preview” of the view offered by the portal 910. Theuser may, for example, view through the portal 910 with full parallax,such that the portal 910 behaves as a type of magnifying lens. Forexample, while looking through the portal 910, the user may view thevirtual environment as if the user has been scaled to the inverse of theportal's scale factor (which affects both the interpupillary distanceand the focal distance) and as if the user has been translated to thereciprocal of the portal's scale factor (1/portal scale factor) of thedistance from the portal 910 to the user's current location.Furthermore, the portal 910 may include an “event horizon” which may bea texture on a plane that is rendered, for example, using additional oneor more cameras (described below) within the virtual environment scenepositioned as described above. In these variations, when “traveling”through the portal 910 after selecting the portal 910 for teleportation,the user's view of the virtual environment may naturally converge withthe user's apparent vantage point during the user's approach to theportal, since the user's vantage point is offset as a fraction of thedistance from the portal (by 1/portal scale factor). Accordingly, theuser may feel as if they are smoothly and naturally stepping intoviewing the virtual environment at the scale factor associated with theselected portal.

As shown in FIG. 9A, in some variations, the portal 910 may be generallycircular. However, in other variations, one or more portals 910 may beany suitable shape, such as elliptical, square, rectangular, irregular,etc. Furthermore, the window view of the virtual environment that isdisplayed in the portal may display the virtual environment at a scalefactor associated with the portal, such that the view of the virtualenvironment displayed in different portals may be displayed at different“zoom” levels (e.g., 1×, 1.5×, 2×, 2.5×, 3×, etc.), thereby alsochanging the scale of the user relative to the environment. The scalefactor of the window view in a portal may also indicate or correspondwith the scale of the view that would be displayed if the user istransported to that portal's vantage point. For example, if a view ofthe virtual environment outside a virtual patient is about 1×, then awindow view of the virtual environment inside the virtual patient may beabout 2× or more, thereby providing a user with more detail of theinternal tissue of the virtual patient. The scale factor may beuser-defined or predetermined by the system (e.g., based on location ofthe portal in the virtual environment). In some variations, the scalefactor may correlate to the displayed size of the portal 910, though inother variations, the scale factor may be independent of the portalsize.

In some variations, a portal 910 may be placed in substantially anyvantage point in the virtual environment that the user desires. Forexample, a portal 910 may be placed anywhere on a virtual ground surfaceof the virtual operating room or on a virtual object (e.g., table,chair, user console, etc.). As another example, as shown in FIG. 9B, aportal 910 may be placed in mid-air at any suitable elevation above thevirtual ground surface. As yet another example, as shown in FIG. 9C, aportal may be placed on or inside a virtual patient, such as portals 910a and 910 b which are placed on the abdomen of a patient and enableviews of the intestines and other internal organs of the virtual patient(e.g., simulated augmented reality). In this example, the virtualpatient may be generated from medical imaging and other information fora real (non-virtual) patient, such that portals 910 a and 910 b mayenable the user to have an immersive view of an accurate representationof the real patient's tissue (e.g., for viewing tumors, etc.), and/orgenerated from internal virtual cameras (described below) placed insidethe patient. In some variations, the system may limit placement of aportal 910 according to predefined guidelines (e.g., only outside thepatient or only inside the patient), which may correspond, for example,to a type of simulated surgical procedure or a training level (e.g.,“beginner” or “advanced” user level) associated with the virtualenvironment.

Such prohibited locations may be indicated to the user by, for example,a visual change in the portal 910 as it is being placed (e.g., changingoutline color, displaying a grayed-out or opaque window view within theport 910 as it is being placed) and/or auditory indications (e.g., beep,tone, verbal feedback). In yet other variations, the system mayadditionally or alternatively include one or more portals 910 placed inpredetermined locations, such as at a virtual user console in thevirtual environment, adjacent the virtual patient table, etc. Suchpredetermined locations may, for example, depend the type of procedure,or be saved as part of a configuration file.

A portal 910 may be viewable from either “side” (e.g., front side andrear side) of the portal. In some variations, the view from one side ofthe portal 910 may be different from an opposite side of the portal 910.For example, when viewed from a first side (e.g., front) of the portal910, the portal may provide a view of the virtual environment with ascale factor and parallax effects as described above, while when viewedfrom a second side (e.g., rear) of the portal 910, the portal mayprovide a view of the virtual environment with a scale factor of aboutone. As another example, the portal may provide a view of the virtualenvironment with a scale factor and parallax effects when viewed fromboth the first side and the second side of the portal.

In some variations, multiple portals 910 may be sequentially linked toinclude a trajectory in the virtual environment. For example, as shownin FIG. 9C, a first-person perspective view of the virtual roboticsurgical environment from a first vantage point may be displayed (e.g.,an immersive view). The user may place a first portal 910 a in a secondvantage point that is different from the first vantage point (e.g.,closer to the virtual patient than the first vantage point) and a firstwindow view of the virtual robotic surgical environment from the secondvantage point may be displayed in the first portal 910 a. Similarly, theuser may place a second portal 910 b in a third vantage point (e.g.,closer to the patient than the first and second vantage points), and asecond window view of the virtual robotic surgical environment may bedisplayed in the second portal 910 b. The user may provide a user inputassociating the first and second portals 910 a and 910 b (e.g., byselection with the handheld controllers, drawing a line between thefirst and second portals with the handheld controllers, etc.) such thatthe first and second portals are sequentially linked, thereby generatinga trajectory between the first and second portals.

In some variations, after multiple portals 910 are linked to generate atrajectory, transportation along the trajectory may not require explicitselection of each sequential portal. For example, once on the trajectory(e.g., at the second vantage point), traveling between linked portalsmay be accomplished by engagement of a trigger, button, touchpad, scrollwheel, other interactive feature of the handheld controller, voicecommand, etc.

Additional portals may be linked in a similar manner. For example, two,three, or more portals may be linked in series to generate an extendedtrajectory. As another example, multiple portals may form “branched”trajectories, where at least two trajectories share at least one portalin common but otherwise each trajectory has at least one portal that isunique to that trajectory. As yet another example, multiple portals mayform two or more trajectories that share no portals in common. The usermay select which trajectory on which to travel, such as by using thehandheld controllers and/or voice command, etc. One or more trajectoriesbetween portals may be visually indicated (e.g., with a dotted line,color coding of portals along the same trajectory, etc.), and suchvisual indication of trajectories may be toggled on and off, such asbased on user preference.

Other portal features may facilitate easy navigation of the trajectoriesbetween portals. For example, a portal may change color when the userhas entered and gone through that portal. As shown in FIG. 9C, inanother example, a portal itself may be displayed with direction arrowsindicating the permissible direction of the trajectory including thatportal. Furthermore, travel along the trajectories may be accomplishedwith an “undo” command (via handheld controllers and/or voice command,etc.) that returns the user to the previous vantage point (e.g.,displays the view of the virtual environment from the previous vantagepoint). In some variations, a home or default vantage point may beestablished (such as according to user preference or system settings) inorder to enable a user to return to that home vantage point quickly witha shortcut command, such as an interactive feature on a handheldcontroller or a voice command (e.g., “Reset my position”). For example,a home or default vantage point may be at a virtual user console oradjacent to the virtual patient table.

The user mode facilitating placement and use of portals, or anotherseparate user mode, may further facilitate deletion of one or moreportals. For example, a portal may be selected for deletion with thehandheld controllers. As another example, one or more portals may beselected for deletion via voice command (e.g., “delete all portals” or“delete portal A”).

Free Navigation

The system may include a user mode that facilitates free navigationaround the virtual robotic surgical environment. For example, asdescribed herein, the system may be configured to detect the user'swalking movements based on sensors in the head-mounted display and/orhandheld controllers, and may correlate the user's movements intorepositioning within a virtual operating room.

In another variation, the system may include a flight mode that enablesthe user to quickly navigate the virtual environment in a “flying”manner at different elevations and/or speeds, and at different angles.For example, the user may navigate in flight mode by directing one ormore handheld controllers and/or the headset in a desired direction forflight. Interactive features on the handheld controller may furthercontrol flight. For example, a directional pad or touchpad may providecontrol for forward, backward, strafing, etc. motions while maintainingsubstantially the same perspective view of the virtual environment.Translation may, in some variations, occur without acceleration, asacceleration may tend to increase the likelihood of simulator sickness.In another user setting, a directional pad or touchpad (or orientationof the headset) may provide control for elevation of the user's apparentlocation within the virtual environment. Furthermore, in somevariations, similar to that described above with respect to portals, ahome or default vantage point within the flight mode may be establishedin order to enable a user to return to that home vantage point quicklywith a shortcut command. Parameters such as speed of flight in responseto a user input may be adjustable by the user and/or set by the systemby default.

Furthermore, in flight mode, the scaling factor of the displayed viewmay be controlled via the handheld controllers. The scaling factor may,for example, affect apparent elevation of the user's location within thevirtual environment. In some variations, the user may use the handheldcontrollers to pull apart two points in the displayed view to zoom outand draw closer two points in the displayed view to zoom in, orconversely pull apart two points in the displayed view to zoom in andraw closer two points in the displayed view to zoom out. Additionallyor alternatively, the user may utilize voice commands (e.g., “increasezoom to 2×”) to change the scaling factor of the displayed view. Forexample, FIGS. 10A and 10B illustrate exemplary views of the virtualenvironment that are relatively “zoomed in” and “zoomed out,”respectively. Parameters such as the speed of the change in scalingfactor, minimum and maximum scaling factor ranges, etc. may beadjustable by the user and/or set by the system by default.

As the user freely navigates the virtual environment in flight mode, thedisplayed view may include features to reduce eye fatigue, nausea, etc.For example, in some variations, the system may include a “comfort mode”in which outer regions of the displayed view are removed as the usernavigates in flight mode, which may, for example, help reduce motionsickness for the user. As shown in FIG. 10C, when in the comfort mode,the system may define a transition region 1030 between an innertransition boundary 1010 and an outer transition boundary 1020 around afocal area (e.g., center) of the user's view. Inside the transitionregion (inside the inner transition boundary 1010), a normal view of thevirtual robotic surgical environment is displayed. Outside thetransition region (outside the outer transition boundary 1020), aneutral view or plain background (e.g., a plain, gray background) isdisplayed. Within the transition region 1030, the displayed view mayhave a gradient that gradually transitions the view of the virtualenvironment to the neutral view. Although the transition region 1030shown in FIG. 10C is depicted as generally circular, with generallycircular inner and outer transition boundaries 1010 and 1020, in othervariations the inner and outer transition boundaries 1010 and 1020 maydefine a transition region 1030 that is elliptical or other suitableshape. Furthermore, in some variations, various parameters of thetransition region, such as size, shape, gradient, etc. may be adjustableby the user and/or set by the system by default.

In some variations, as shown in FIG. 11, the user may view the virtualrobotic surgical environment from a dollhouse view that allows the userto view the virtual operating room from an overhead vantage point, witha top-down perspective. In the dollhouse view, the virtual operatingroom may be displayed at a smaller scale factor (e.g., smaller thanlife-size) on the display, thereby changing the scale of the userrelative to the virtual operating room. The dollhouse view may providethe user with additional contextual awareness of the virtualenvironment, as the user may view the entire virtual operating room atonce, as well as the arrangement of its contents, such as virtualequipment, virtual personnel, virtual patient, etc. Through thedollhouse view, for example, the user may rearrange virtual objects inthe virtual operating room with fuller contextual awareness. Thedollhouse view may, in some variations, be linked in a trajectory alongwith portals and/or snap points described above.

Environment View Rotation

In some variations, the system may include a user mode that enables theuser to navigate the virtual robotic surgical environment by moving thevirtual environment around his or her current vantage point. Theenvironment view rotation mode may offer a different manner in which theuser may navigate the virtual environment, such as by “grasping” andmanipulating the environment as if it were an object. As the usernavigates through the virtual environment in such a manner, a “comfortmode” similar to that described above may additionally be implemented tohelp reduce simulation-related motion sickness. For example, in anenvironment view rotation mode, the user may rotate a displayed scenearound a current vantage point by selecting and dragging the view of thevirtual environment around the user's current vantage point. In otherwords, in the environment view rotation mode, the user's apparentlocation in the virtual environment appears fixed while the virtualenvironment may be moved. This is contrast to other modes, such as, forexample, flight mode described above, in which generally the environmentmay appear fixed while the user moves. Similar to the scaling factoradjustments described above for flight mode, in the environment viewrotation mode, the scaling factor of the displayed view of theenvironment may be controlled by the handheld controllers and/or voicecommands (e.g., by using the handheld controllers to select and pullapart two points in the displayed view to zoom in, etc.).

For example, as shown in FIG. 15, in one exemplary variation of a method1500 for operating in an environment view rotation mode, the user mayactivate a user input method (1510) such as on a handheld controller(e.g., a button or trigger or other suitable feature) or any suitabledevice. In some variations, one handheld controller (1520) may bedetected upon activation of the user input method. The original positionof the handheld controller at the time of activation may be detected andstored (1522). Thereafter, as the user moves the handheld controller(e.g., while continuing to activate the user input method), the currentposition of the handheld controller may be detected (1524). A vectordifference between the original (or previous) position and the currentposition of the handheld controller may be calculated (1526), andposition of the vantage point of the user may be adjusted (1528) basedat least partially on the calculated vector difference, thereby creatingan effect that makes the user feel they are “grabbing” and dragging thevirtual environment around.

In some variations, two handheld controllers (1520′) may be detectedupon activation of the user input method. The original positions of thehandheld controllers may be detected (1522′), and a centerpoint and anoriginal vector between the original positions of the handheldcontrollers (1523′) may be calculated and stored. Thereafter, as theuser moves one or more both handheld controllers (e.g., while continuingto activate the user input method), the current positions of thehandheld controllers may be detected (1524′) and used to form the basisfor a calculated vector difference between original and current vectorsbetween handheld controllers (1528′). The position and/or orientation ofthe vantage point of the user may be adjusted (1528′), based on thecalculated vector difference. For example, the orientation or rotationof the displayed view may be rotated around the centerpoint between thehandheld controller locations, thereby creating an effect that makes theuser feel they are “grabbing” and dragging the environment around.Similarly, the scale of the display of the virtual environment (1529′)may be adjusted based on the calculated difference in distance betweenthe two handheld controllers, thereby creating an effect that makes theuser feel they are “grabbing” and zooming in and out of the displayedview of the virtual environment.

Although the above-described user modes are described separately, itshould be understood that aspects of these modes characterize exemplaryways that a user may navigate the virtual robotic surgical environment,and may be combined in a single user mode. Furthermore, some of theseaspects may be seamlessly linked. For example, an overhead vantage pointgenerally associated with flight mode may be sequentially linked withone or more portals in a trajectory. Even further, in some variations, avantage point or displayed view of the virtual environment (e.g., asadjusted via one or more of the above user modes) may be linked to atleast one default vantage point (e.g., default in position, orientation,and/or scale). For example, by activating a user input (e.g., on ahandheld controller, foot pedal, etc.), a user may “reset” the currentvantage point to a designated or predetermined vantage point in thevirtual environment. The user's current vantage point may, for example,be gradually and smoothly animated in order to transition to defaultvalues of position, orientation, and/or scale.

Supplemental Views

In some variations, an exemplary user mode or modes of the system maydisplay one or more supplemental views of additional information to auser, such as overlaid over or inset in the primary, first-personperspective view of the virtual robotic surgical environment. Forexample, as shown in FIG. 12, a heads-up display 1210 (HUD) may providea transparent overlay over a primary first-person perspective view ofthe virtual environment. The HUD 1210 may toggled on and off, therebyallowing the user to control whether to display the HUD 1210 at anyparticular time. Supplemental views of additional information, such asthat described below, may be placed onto the HUD such that the user mayobserve the supplemental views without looking away from the primaryview. For example, the supplemental views may “stick” to the HUD 1210and move with the user's head movement such that the supplemental viewsare always in the user's field of view. As another example, supplementalviews of additional information may be loosely fixed to the HUD 1210, inthat the supplemental views may be small or at least partially hiddenoff-screen or in peripheral vision when the user's head is generallyfacing forward, but minor or slight head movement to one side may expandand/or bring one or more supplemental views into the user's field ofview. The one or more supplemental views may be arranged on the HUD 1210in a row, grid, or other suitable arrangement. In some variations, theHUD 1210 may include predetermined “snap” points to which thesupplemental views (e.g., camera views, etc.) are positioned. Forexample, the user may select a supplemental view on the HUD 1210 forcloser inspection, then replace the supplemental view on the HUD 1210 bydragging it generally in the direction of a snap point, whereupon thesupplemental view may be drawn to and fixed at the snap point withoutneeding to be precisely placed there by the user.

As another example, in a virtual command station mode, one or moresupplemental views may be displayed in a virtual space with one or morecontent windows or panels arranged in front of the user in the virtualspace (e.g., similar to a navigable menu). For example, as shown in FIG.13, multiple content windows (e.g., 1310 a, 1310 b, 1310 c, and 1310 d)may be positioned in a semi-circular arrangement or other suitablearrangement for display to a user. The arrangement of the contentwindows may be adjusted by the user (e.g., using handheld controllerswith their graphical representations 230′ to select and drag or rotatecontent windows). The content windows may display, for example, anendoscope video feed, a portal view, a “stadium” overhead view of thevirtual operating room, patient data (e.g., imaging), other camera orpatient information views such as those described herein, etc. Byviewing multiple panels simultaneously, the user may be able tosimultaneously monitor multiple aspects of the virtual operating roomand/or with the patient, thereby allowing the user to have anoverarching and broader awareness of the virtual environment. Forexample, the user may become aware of and then respond more quickly toany adverse events in the virtual environment (e.g., simulated negativereactions of the virtual patient during a simulated surgical procedure).

Furthermore, the virtual command station mode may enable a user toselect any one of the content windows and become immersed in thedisplayed content (e.g., with a first-person perspective). Such a fullyimmersive mode may temporarily dismiss the other content windows, or mayminimize (e.g., be relegated to a HUD overlaid over the selectedimmersive content). As an illustrative example, in the virtual commandstation mode, the system may display multiple content windows includingan endoscopic camera video feed showing the inside of a virtualpatient's abdomen. The user may select the endoscopic camera video feedto become fully immersed in the virtual patient's abdomen (e.g., whilestill manipulating robotic arms and instruments attached to the arms).

Camera Views

In some variations, a user mode may enable the user to place a virtualcamera in a selected vantage point in the virtual environment, and awindow view of the virtual environment from the selected vantage pointmay be displayed in the HUD such that the user may simultaneously viewboth his first-person perspective field of view and the camera view (theview provided by the virtual camera) that may update in real time. Avirtual camera may be placed in any suitable location in the virtualenvironment (e.g., inside or outside the patient, overhead the patient,overhead the virtual operating room, etc.). For example, as shown inFIG. 12, the user may place a virtual camera 1220 (e.g., using objectgripping as described above) near the pelvic region of a virtual patientand facing the patient's abdomen so as to provide a “virtual video feed”of the patient's abdomen. Once placed, a virtual camera 1220 may besubsequently repositioned. A camera view (e.g., a circular inset view,or window of any suitable shape) may be placed on the HUD as a windowview showing the virtual video feed from the vantage point of thevirtual camera 1220. Similarly, multiple virtual cameras may be placedin the virtual environment to enable multiple camera views to be shownon the HUD. In some variations, a predetermined arrangement of one ormore virtual cameras may be loaded, such as part of a configuration filefor the virtual reality processor to incorporate into the virtualenvironment.

In some variations, the system may offer a range of different kinds ofvirtual cameras, which may provide different kinds of camera views. Oneexemplary variation of a virtual camera is a “movie” camera that isconfigured to provide a live virtual feed of the virtual environment(e.g., movie camera view 1212 in FIG. 12). Another exemplary variationof a virtual camera is an endoscopic camera which is attached to avirtual endoscope to be placed in a virtual patient. In this variation,the user may, for example, virtually perform a technique for introducingthe virtual endoscope camera into the virtual patient and subsequentlymonitor the internal workspace inside the patient by viewing the virtualendoscopic video feed (e.g., endoscopic camera view 1214 in FIG. 12). Inanother exemplary variation, the virtual camera may be a wide-angle(e.g., 360-degree, panoramic, etc.) camera that is configured to providea larger field of view of the virtual environment. In this variation,the window camera view may, for example, be displayed as a fish-eye orgenerally spherical display.

Various aspects of the camera view may be adjusted by the user. Forexample, the user may adjust the location, size, scale factor, etc. ofthe camera view (e.g., similar to adjustments of portals as describedabove). As another example, the user may select one or more filters orother special imaging effects to be applied to the camera view.Exemplary filters include filters that highlight particular anatomicalfeatures (e.g., tumors) or tissue characteristics (e.g., perfusion) ofthe virtual patient. In some variations, one or more virtual cameras maybe deselected or “turned off” (e.g., have the virtual camera and/orassociated camera view selectively hidden) or deleted, such as if thevirtual camera or its associated camera view is obstructing the user'sview of the virtual environment behind the virtual camera or cameraview.

In some variations, a camera view may function similarly to a portal(described above) to enable the user to navigate quickly around thevirtual environment. For example, with reference to FIG. 12, a user mayselect the camera view 1212 (e.g., highlight or grab and pull the cameraview 1212 toward himself) to be transported to the vantage point of thecamera view 1212.

Patient Data Views, Etc.

In some variations, a user mode may enable display of patient data andother information on the HUD or another suitable location in thedisplay. For example, patient imaging information (e.g., ultrasound,X-ray, Mill, etc.) may be displayed in a supplemental display, overlaidover the patient (e.g., as simulated augmented reality). A user may, forexample, view patient images as reference while interacting with thevirtual patient. As another example, patient vitals (e.g., heartrate,blood pressure, etc.) may be displayed to the user in a supplementalview.

In another variation, a user mode may enable display of other suitableinformation, such as training videos (e.g., exemplary surgicalprocedures recorded from a previous procedure), video feed from a mentoror trainer surgeon, etc. providing guidance to a user.

Virtual Reality System Applications

Generally, the virtual reality system may be used in any suitablescenario in which it is useful to simulate or replicate a roboticsurgical environment. In some variations, the virtual reality system maybe used for training purposes, such as allowing a surgeon to practicecontrolling a robotic surgical system and/or to practice performing aparticular kind of minimally-invasive surgical procedure using a roboticsurgical system. The virtual reality system may enable a user to betterunderstand the movements of the robotic surgical system in response touser commands, both inside and outside the patient. For example, a usermay don a head-mounted display under supervision of a mentor or trainerwho may view the virtual reality environment alongside the user (e.g.,through a second head-mounted display, through an external display,etc.) and guide the user through operations of a virtual roboticsurgical system within the virtual reality environment. As anotherexample, a user may don a head-mounted display and may view, asdisplayed on the immersive display (e.g., in a content window, the HUD,etc.) a training-related video such as a recording of a previouslyperformed surgical procedure.

As another example, the virtual reality system may be used for surgicalplanning purposes. For example, a user may operate the virtual realitysystem to plan surgical workflow. Configuration files of virtual objects(e.g., robotic surgical system including arms and tool drivers, userconsole, end effectors, other equipment, patient bed, patient,personnel, etc.) may be loaded into a virtual robotic surgicalenvironment as representative of actual objects that will be in theactual (i.e., non-virtual, or real) operating room. Within the virtualreality environment, the user may adjust features of the virtualoperating room, such as positioning the user console, patient bed, andother equipment relative to one another in a desired arrangement. Theuser may additionally or alternatively use the virtual reality system toplan aspects of the robotic surgical system, such as selecting numberand location of ports for entry of the surgical instruments, ordetermining optimum number and position/orientation (e.g., mountinglocation, arm pose, etc.) of robotic arms for a procedure, such as forminimizing potential collisions between system components during thesurgical procedure. Such virtual arrangements may be based on, forexample, trial-and-error, previous setups for similar surgicalprocedures and/or similar patients, etc. In some variations, the systemmay additionally or alternatively propose virtual arrangements selectedbased on machine learning techniques applied to datasets ofpreviously-performed surgical procedures for various kinds of patients.

As yet another example, the virtual reality system may be used for R&Dpurposes (e.g.. simulation). For example, a method for designing arobotic surgical system may include generating a virtual model of arobotic surgical system, testing the virtual model of the roboticsurgical system in a virtual operating room environment, modifying thevirtual model of the robotic surgical system based on the testing, andbuilding the robotic surgical system based on the modified virtualmodel. Aspects of the virtual model of the robotic surgical system thatmay be tested in the virtual operating room environment include physicalcharacteristics of one or more components of the robotic surgical system(e.g., diameter or length of arm links). For example, a virtual model ofa particular design of a robotic arm may be built and implemented in avirtual environment, where the virtual model may be tested with respectto particular arm movements, surgical procedures, etc. (e.g., test forlikelihood of collision between the robotic arm and other objects).Accordingly, a design of a robotic arm (or similarly, any othercomponent of the robotic surgical system) may be at least initiallytested by testing a virtual implementation of the design, rather thantesting a physical prototype, thereby accelerating the R&D cycle andreducing costs.

Other aspects that may be tested include functionality of one or morecomponents of the robotic surgical system (e.g., control modes of acontrol system). For example, as described above, a virtual operatingenvironment application may pass status information to a kinematicsapplication, and the kinematics application may generate and passcommands based on control algorithms, where the virtual realityprocessor may use the commands to cause changes in the virtual roboticsurgical environment (e.g., move a virtual robotic arm in a particularway in accordance with relevant control algorithms). As such, softwarecontrol algorithms may be embodied in a virtual robotic system fortesting, refinement, etc. without requiring a physical prototype of therelevant robotic component, thereby conserving R&D resources andaccelerating the R&D cycle.

In another example, the virtual reality system may be used to enablemultiple surgeons to collaborate in the same virtual realityenvironment. For example, multiple users may don head-mounted displaysand interact with each other (and with the same virtual robotic system,the same virtual patient, etc.) in the virtual reality environment. Theusers may be physically in the same room or general location, or may beremote from one another. For example, one user may be tele-mentoring theother as they collaborate to perform a surgical procedure on the virtualpatient.

Specific illustrative exemplary applications of the virtual realitysystem are described in further detail below. However, it should beunderstood that applications of the virtual reality system are notlimited to these examples and general application scenarios describedherein.

Example 1—Over the Bed

A user may use the virtual reality system to simulate an over-the-bedscenario in which he is adjacent to a patient bed or table and operatingboth a robotic surgical system and a manual laparoscopic tool. Suchsimulation may be useful for training, surgical planning, etc. Forexample, the user may staple tissue in a target segment of a virtualpatient's intestine using both a virtual robotic tool and a virtualmanual tool.

In this example, the user dons a head-mounted display providing animmersive view of a virtual reality environment, and may use handheldcontrollers to navigate within the virtual reality environment to beadjacent to a virtual patient table on which a virtual patient lies. Aproximal end of a virtual robotic arm is attached to the virtual patienttable, and a distal end of the virtual robotic arm supports a virtualtool driver actuating virtual forceps that are positioned within theabdomen of the virtual patient. A virtual manual laparoscopic staplertool is passed through a virtual cannula and having a distal endpositioned within the abdomen of the virtual patient. Additionally, anendoscopic camera is positioned within the abdomen of the virtualpatient, and provides a virtual camera feed showing the surgicalworkspace within the abdomen of the virtual patient (including patienttissue, virtual robotically-controlled forceps, and a virtual manuallaparoscopic stapler tool).

The user continues to view the virtual environment through the immersivedisplay in the head-mounted display, as well as the virtual endoscopiccamera feed displayed in a window view in a heads-up display overlaid inthe user's field of view. The user holds in one hand a handheldcontroller that is configured to control the robotically-driven virtualforceps. The user holds in another hand a laparoscopic hand controllerthat is configured to control the virtual manual laparoscopic staplertool, with the laparoscopic hand controller passing through a cannulamounted in a mock patient body made of foam. The laparoscopic handcontroller is calibrated to correspond to the virtual manuallaparoscopic stapler tool. The user manipulates the handheld controllerto operate the robotically-controlled forceps to manipulate theintestine of the virtual patient and uncover a target segment of theintestine. With the target segment of the intestine exposed andaccessible, the user manipulates the laparoscopic hand controller toapply virtual staples to the target segment via the virtual manuallaparoscopic stapler tool.

Example 2—Collision Resolution from User Console

When using the virtual reality system, a user may desire to resolvecollisions between virtual components of the virtual robotic surgicalsystem, even though the user may not be adjacent the colliding virtualcomponents (e.g., the user may be seated at a distance away from thevirtual patient table, such as at a virtual user console). In thisexample, the user dons a head-mounted display providing an immersiveview provided by a virtual endoscope placed inside an abdomen of avirtual patient. Proximal ends of two virtual robotic arms are attachedto separate locations on a virtual patient table, on which the virtualpatient lies. Distal ends of the virtual robotic arms support respectivetool drivers actuating virtual forceps that are positioned within theabdomen of the virtual patient. The user manipulates the handheldcontrollers to operate the two robotically-controlled virtual forceps,which manipulate virtual tissue within the virtual patient. Thismovement may cause a collision involving at least one of the virtualrobotic arms (e.g., a virtual robotic arm may be posed so as to create acollision with itself, the virtual robotic arms may be posed so as tocreate a collision with each other, a virtual robotic arm may be posedso as to create a collision with the patient or nearby obstacle, etc.).

The virtual reality system detects the collision based on statusinformation of the virtual robotic arms, and alerts the user regardingthe collision. The system displays an overhead or other suitable viewfrom a suitable vantage point of the virtual robotic surgical system,such as in a window view (e.g., picture-in-picture view). The locationof the collision is highlighted in the displayed window view, such as byoutlining the affected colliding components with red or anothercontrasting color. Alternatively, the user may detect the collisionhimself by monitoring a camera video feed from a virtual camera placedoverhead the virtual patient table.

Upon becoming aware of the collision, the user may zoom out or adjustthe scale of his immersive view of the virtual reality environment. Theuser may engage an arm repositioning control mode that locks theposition and orientation of the virtual forceps within the patient.Using the handheld controllers in an object gripping user mode, the usermay grab onto virtual touchpoints on the virtual robotic arms andreposition (repose) the virtual robotic arms so as to resolve thecollision while the control mode maintains the position and orientationof the virtual forceps during the arm repositioning. Once the virtualrobotic arms are repositioned such that the collision is resolved, theuser may zoom back into the previous vantage point, disengage the armrepositioning control mode, and resume using the handheld controllers tooperate the virtual forceps within the virtual patient.

Example 3—Coordinated Relocation of Multiple Surgical Instruments fromUser Console

When using the virtual reality system, a user may find it useful to staysubstantially in an endoscopic view and relocating multiple virtualsurgical instruments (e.g., end effectors, cameras) as a group ratherthan individually within the virtual patient, thereby saving time, aswell as making it easier for the user to maintain contextual awarenessof the instruments relative to the virtual patient's anatomy. In thisexample, the user dons a head-mounted display providing an immersiveview provided by a virtual endoscope placed inside an abdomen of avirtual patient. Proximal ends of two virtual robotic arms are attachedto separate locations on a virtual patient table, on which the virtualpatient lies. Distal ends of the virtual robotic arm support respectivetool drivers actuating virtual forceps that are positioned in the pelvicarea of the virtual patient. The user may manipulate handheldcontrollers to operate the virtual forceps.

The user may wish to move the virtual endoscope and the virtual forcepsto another target region of the virtual patient's abdomen, such as thespleen. Rather than move each surgical instrument individually, the usermay engage a coordinated relocation mode. Once this mode is engaged, theendoscopic camera view zooms out along the axis of the endoscope to adistance sufficient to allow the user to view the new target region(spleen). A spherical indicator is displayed at the distal end of theendoscope that encapsulates the distal end of the virtual endoscope andthe distal ends of the virtual forceps. The user manipulates at leastone handheld controller to withdraw the virtual endoscope and thevirtual forceps away from the surgical workspace (e.g., until the usercan see the distal end of the virtual cannula in the virtual endoscopicview), then grab and move the spherical indicator from the pelvic areato the spleen. Once the user finalizes the new target region by movingthe spherical indicator to the new target region, the virtual endoscopeand virtual forceps automatically travel to the new target region andthe virtual endoscopic camera view zooms to show the new target region.Throughout this relatively large-scale move, the user views the virtualenvironment with substantially an endoscopic view of the virtualenvironment, thereby enabling the user to maintain awareness of thevirtual patient's anatomy instead of transferring his focus betweeninstrument and anatomy.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the invention arepresented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed; obviously, many modifications and variations are possible inview of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical applications, they thereby enable others skilled in the art tobest utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the following claims and their equivalents define thescope of the invention.

1. (canceled)
 2. A virtual reality system for simulating a roboticsurgical environment, the system comprising: a virtual reality processorconfigured to generate a virtual robotic surgical environment comprisingat least one virtual robotic arm and at least one virtual manuallaparoscopic tool; and a handheld device communicatively coupled to thevirtual reality processor for manipulating the at least one virtualmanual laparoscopic tool in the virtual robotic surgical environment,the handheld device including a handheld portion, and a tool featurethat is removable from the handheld portion, the tool feature beingrepresentative of at least a portion of a manual laparoscopic tool. 3.The virtual reality system of claim 2, wherein the handheld device isused for manipulation of the at least one virtual robotic arm roboticarm when the tool feature is removed from the handheld portion of thehandheld device.
 4. The virtual reality system of claim 2, wherein thehandheld device is used for manipulation of a virtualrobotically-controlled tool when the tool feature is removed from thehandheld portion of the handheld device.
 5. The virtual reality systemof claim 2, wherein the tool feature comprises a tool shaft and a shaftadapter for coupling the tool shaft to the handheld portion of thesecond handheld device.
 6. The virtual reality system of claim 5,wherein the shaft adapter comprises fasteners.
 7. The virtual realitysystem of claim 2, wherein the handheld portion comprises an interactivefeature that actuates a function of the virtual manual laparoscopic toolin response to engagement of the interactive feature by a user.
 8. Thevirtual reality system of claim 7, wherein the interactive featurecomprises a trigger.
 9. The virtual reality system of claim 2, whereinthe virtual manual laparoscopic tool is a virtual manual laparoscopicstapler.
 10. The virtual reality system of claim 2, wherein a trigger onthe handheld portion controls firing of a virtual staple.
 11. Thevirtual reality system of claim 2, further comprising a patientsimulator comprising a cannula arranged to receive at least a portion ofthe tool feature of the handheld device.
 12. A computer-implementedmethod for simulating a robotic surgical environment in a virtualreality system, the method comprising: generating, by one or moreprocessors, a virtual robotic surgical environment comprising at leastone virtual robotic arm and a virtual manual laparoscopic tool; couplinga handheld device communicatively to the processors for manipulating thevirtual manual laparoscopic tool in the virtual robotic surgicalenvironment, the handheld device including a handheld portion and a toolfeature that is removable from the handheld portion, the tool featurebeing representative of at least a portion of a manual laparoscopictool; and simulating an over-the-bed surgery in the virtual realitysystem based on a user's input with the handheld device.
 13. The methodof claim 12, wherein the handheld device is used for manipulation of theat least one virtual robotic arm robotic arm when the tool feature isremoved from the handheld portion of the handheld device.
 14. The methodof claim 12, wherein the handheld device is used for manipulation of avirtual robotically-controlled tool when the tool feature is removedfrom the handheld portion of the handheld device.
 15. The method ofclaim 12, wherein the tool feature comprises a tool shaft and a shaftadapter for coupling the tool shaft to the handheld portion of thesecond handheld device.
 16. The method of claim 15, wherein the shaftadapter comprises fasteners.
 17. The method of claim 12, wherein thehandheld portion comprises an interactive feature that actuates afunction of the virtual manual laparoscopic tool in response toengagement of the interactive feature by a user.
 18. The method of claim17, wherein the interactive feature comprises a trigger.
 19. The methodof claim 12, wherein the virtual manual laparoscopic tool is a virtualmanual laparoscopic stapler.
 20. The method of claim 12, wherein atrigger on the handheld portion controls firing of a virtual staple. 21.The method of claim 12, further comprising a patient simulatorcomprising a cannula arranged to receive at least a portion of the toolfeature of the handheld device.